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PREFACE 


This  report  concludes  the  Queueing  Methodology  study  that  has  been 
directed  toward  the  development  of  dynamic  models  for  evaluating  anti¬ 
submarine  warfare  (ASW)  system  effectiveness.  The  project  was  sponsored 
by  the  Director,  Naval  Analysis  Programs,  of  the  Office  of  Naval  Research 
(ONR) .  Mr.  R.  H.  Dickman  was  the  ONR  Project  Scientific  Officer.  The 
research  effort  was  performed  jointly  by  Stanford  Research  Institute's 
(SRI)  Naval  Warfare  Research  Center  (NWRC),  Mr.  L.  J.  Low,  Director, 
and  the  Information  Sciences  Laboratory,  Mr.  D.  R.  Brown,  Director. 

Mr.  W.  H.  Frye  of  NWRC  was  the  project  leader  for  that  part  of  the  study 
leading  to  this  report. 

The  first  report  on  the  project,  "Dynamic  Analysis  of  ASW  Effective- 

J|e 

ness — A  Queueing  Approach,"  was  published  in  March  1972. 1  This  report 
now  concludes  the  project  with  the  presentation  of  the  research  results 
that  complement  the  results  from  the  first  report. 

The  following  SRI  personnel  contributed  to  this  final  report: 

William  H.  Frye  (NWRC,  project  leader) 

Andrew  J.  Korsak  (Information  Sciences  Laboratory). 

In  addition,  computer  programming  assistance  was  provided  by  D.  L. 
Alderman  and  D.  G.  Ayers  of  NWRC. 


* 

References  are  listed  at  the  end  of  this  report. 
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I  INTRODUCTION  AND  SUMMARY 


A.  Research  Objectives 

The  objective  of  the  Queueing  Methodology  study  was  to  develop  new 
methodology  for  evaluating  ASW  system  effectiveness  in  dynamic  contexts. 

It  should  be  specifically  noted  that  the  study  was  not  directed  toward 
the  solution  of  any  particular  real-world  ASW  problem.  Rather,  the  prob¬ 
lem  addressed  is  how  to  adequately  analyze  and  measure  ASW  system  effec¬ 
tiveness  in  complex,  dynamic,  tactical  situations. 

To  understand  the  approach  taken  in  this  research  effort,  some  back¬ 
ground  in  the  nature  of  naval  effectiveness  modeling  is  needed.  Models, 
modeling  efforts,  and  effectiveness  analysis  can  be  divided  into  two 
categories:  analytical  or  operational.  Roughly  speaking,  analytical 

models  are  usually  constructed  on  the  assumption  that  the  model  struc¬ 
ture  and  parameters  are  known  or  will  somehow  be  estimated  by  the  time 
the  model  is  used  on  a  real  problem.  Parameter  estimation,  and  the  struc¬ 
turing  of  the  model  so  that  its  parameters  can  be  estimated,  is  usually 
of  secondary  concern  in  such  modeling  efforts.  In  contrast,  operational 
models  focus  on  the  data  available  (or  to  become  available),  and  the 
parameters  are  estimated  from  these  data. 

Most  ASW  effectiveness  modeling  efforts  are  analytical,  and  in  many 
respects  analytical  models  are  more  highly  developed  than  operational 
models.  War  games  and  Monte  Carlo  simulation,  which  are  common  forms  of 
analytical  models,  have  advanced  considerably  in  many  technical  respects 
over  the  past  decade  or  so.  In  particular,  complex  dynamic  tactical 
situations  are  now  routinely  modeled  by  analytical  means.  Whatever  the 
advantages  of  these  advanced  analytical  models,  they  suffer  from  many 
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inherent  weaknesses,  particularly  the  difficulties  in  estimating  parameters, 
or  more  generally  in  demonstrating  model  validity. 

On  the  other  hand,  operational  models — which  are  nearly  always  some 
form  of  conditional  probability  model — have  their  own  weaknesses.  Al¬ 
though  parameter  estimation  (and  therefore  model  validity)  is  not  a  major 
problem  (by  virtue  of  its  very  construction),  the  operational  models  seem 
to  have  limited  usefulness  and  flexibility. 

Both  analytical  and  operational  models  were  studied  during  the  re¬ 
search  effort.  The  first  part  of  the  study  was  devoted  to  analytical 

1  * 

queueing  models  that  are  presented  in  detail  in  an  interim  report. 
Throughout  this  part  of  the  study,  the  model  parameters  were  assumed  to 
be  known  and  queueing  models  were  formulated  for  making  ASW  effectiveness 
calculations.  As  the  study  progressed,  however,  it  became  evident  that 
these  calculations  could  be  handled  more  conveniently  by  Markov  methods 
than  by  queueing  methods.  Moreover,  the  interim  report  also  clearly 
shows  that  Markov  methods  could  conveniently  be  used  to  calculate  such 
dynamic  quantities  as  the: 

•  Expected  number  of  contacts  lost  while  in  the  ASW  system 
(over  a  given  period  of  time). 

•  Probability  of  missing  a  true  contact  (failure  to  detect) 
for  a  given  level  of  contact  activity  (false  and  true 
contacts) . 

•  Expected  time  to  detection. 

•  Expected  time  from  detection  to  kill. 

Several  limitations  of  the  queueing  models  emerged  as  the  study  pro¬ 
gressed.  On  the  one  hand,  parameter  estimation  proved  to  be  as  formidable 
a  problem  for  the  queueing  models  as  it  is  for  dynamic  Monte  Carlo  simu¬ 
lation  models.  On  the  other  hand,  a  major  difficulty  arose  from  computa¬ 
tional  considerations:  The  more  complex  queueing  models  required  so  much 
* 

References  are  listed  at  the  end  of  the  report. 
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computer  time  that  direct  simulation  became  more  attractive  for  acquiring 
numerical  results.  The  conclusion  that  the  queueing  models  chosen  for 
study  were  at  a  dead  end  therefore  became  inescapable,  and  a  reevaluation 
of  research  objectives  was  accordingly  made.  The  particular  queueing 
models  that  had  been  studied  in  this  phase  proved  to  be  valueless  for  the 
next  phase.  Nevertheless,  the  experience  gained  with  Markov  methodology 
(including  the  formulation  of  ASW  problems  in  terms  of  states  and  transi¬ 
tions,  as  well  as  formulas  and  numerical  techniques  for  obtaining  answers) 
proved  to  be  valuable  during  the  remainder  of  the  study. 

Research  effort  during  the  second  part  of  the  study  was  concentrated 
on  a  complementary  aspect  of  effectiveness  modeling;  namely,  model  struc¬ 
turing  and  parameter  estimation.  This  led  to  consideration  of  dynamic 
operational  models  instead  of  dynamic  analytical  models.  Examination  of 
the  classical  conditional  probability  models  such  as  the  weapon  system 
effectiveness  (WSE)  models  was  considered  as  the  starting  point  for  this 
part  of  the  research  effort.  Thus,  the  emphasis  in  the  second  part  of 
the  study  was  on  the  development  of  methods  for  structuring  dynamic  models 
and  estimating  their  parameters  rather  than  methods  for  calculating 
measures  of  effectiveness  when  the  model  parameters  are  known.  More 
specifically,  the  objectives  for  this  part  of  the  study  were  to: 

•  Analyze  conditional*  probability  effectiveness  models  and 
determine  their  limitations. 

•  Generalize  conditional  probability  effectiveness  models 
(using  Markov  methods  if  they  proved  appropriate)  to: 

-  Remove  some  of  the  limitations  on  conditional  probability 
models;  in  particular,  determine  ways  to  make  them  dynamic 
by  incorporating  false  contacts. 

-  Include,  when  appropriate,  other  dynamic  aspects  of  the 
tactical  problem  being  modeled. 

•  Develop  methods  from  simulated  data  and  Fleet  operational 
data  for  estimating  parameters  in  the  dynamic  operational 
models. 
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•  Define  and  investigate  ways  to  formulate  new  state  spaces, 
and  investigate  potential  ASW  effectiveness  applications  if 
operational  Markov  models  proved  appropriate. 

•  Investigate  methods  for  reducing  the  variance  of  effective¬ 
ness  measure  estimates  and  for  using  variance  estimates  in 
exercise  design. 

* 

B.  Research  Findings 


This  section  summarizes  research  findings,  and  Section  C  summarizes 
research  activities  for  the  second  part  of  the  study.  Every  attempt  has 
been  made  to  separate  what  was  found  from  what  was  done.  In  many  respects, 
however,  the  findings  are  rather  general  and  may  often  be  difficult  to 
relate  to  research  activities  without  a  detailed  study  of  the  body  of  the 
report. 

During  the  final  phase  of  the  Queueing  Methodology  study,  research 
findings  ascertained: 

y  •  Queueing  theory  itself  has  no  direct,  unique  applicability 
to  the  analysis  of  ASW  system  effectiveness.  However,  some 
of  the  conceptual  and  computational  methods — particularly 
Markov  methods — often  used  in  the  analysis  of  queueing  models 
are  well-suited  to  analyzing  effectiveness  in  today's  complex 
tactical  environments. 

•  Conditional  probability  models,  the  most  commonly  used  opera¬ 
tional  models,  have  many  limitations. 

-  They  are  static  models,  not  dynamic.  One  consequence  of 
this  is  that  they  are  defined  for  only  a  limited  number 
of  tactical  situations,  usually  one-on-one  engagements. 

-  They  have  limited  utility,  flexibility,  and  predictive 
ability  in  new  situations. 

-  Meaningful  sensitivity  analyses  cannot  be  performed. 

-  They  do  not  explicitly  include  factors  such  as  false  con¬ 
tacts  that  degrade  ASW  system  capability. 

-  They  use  only  a  limited  portion  of  the  data  available  for’ 
parameter  estimation. 
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-  They  have  few — if  any — environmental  or  operational 
variables,  and  causative  variables  may  be  difficult  to 
isolate. 

Markov  (and  semi-Markov)  methods  are  not  only  be  used  in¬ 
creasingly  in  analytical  ASW  effectiveness  modeling  but  also 
appear  to  have  considerable  potential  for  operational  effec¬ 
tiveness  modeling  in  dynamic  ASW  environments. 

A  Markov  chain  model  can  be  regarded  as  the  natural  general¬ 
ization  of  a  conditional  probability  model.  This  generaliza¬ 
tion  can  remove  many  of  the  limitations  on  traditional, 
conditional  probability  models. 

Operational  Markov  models  have  both  advantages  and  disad¬ 
vantages.  The  principal  advantage  is  that  operationally 
meaningful  measures  of  effectiveness  can  be  defined  for  a 
large  variety  of  dynamic  tactical  contexts.  The  four 
principal  overall  effectiveness  measures  that  Markov  models 
can  define  are: 

-  The  probability  of  mission  success. 

-  The  average  time  until  mission  failure  (or  success). 

-  The  probability  that  mission  failure  does  not  occur  by  a 
specified  time. 

-  The  probability  of  occupying  a  designated  state  at  a  random 
time. 

Many  measures  of  effectiveness  used  in  naval  analysis  studies 
are  equivalent  to  one  of  these  four  measures.  A  wide  variety 
of  missions  and  tactical  situations  can  be  treated  by  suitably 
defining  success  and  failure  states. 

The  principal  disadvantage  is  that  some  properties  of  opera¬ 
tional  Markov  models  were  found  to  be  rather  "sterile." 

That  is,  the  output  effectiveness  measure  sometimes  too 
perfectly  reflects  the  data  inputs  used  to  estimate  parameters. 
The  four  effectiveness  measures  differ  markedly  in  this  re¬ 
spect;  specifically,  when  data  from  "complete"  exercises  are 
used  to  estimate  the  transition  probabilities  in  a  Markov 
chain  model,  it  was  found  that: 

-  The  estimated  probability  of  success  obtained  from  Markov 
calculations  is  identical  to  the  ratio  of  the  number  of 
successes  to  the  number  of  exercises. 


The  estimated  conditional  mean  time  until  failure  obtained 
from  Markov  calculations  is  identical  to  the  ratio 


E  (times  until  failure) 
total  number  of  failures 


provided  that  only  the  path  data  leading  to  failure  are 
used  to  estimate  the  Markov  parameters. 

These  assertions  can  be  translated  into  implications  con¬ 
cerning  the  statistical  improvability  of  the  measures: 

-  The  variance  of  the  estimate  of  Prob(success)  is  not  re¬ 
ducible  by  using  a  Markov  model. 

-  The  variance  of  the  Markov  estimate  of  the  average  time 

to  fail  (given  that  failure  ultimately  occurs)  is  reducible 
by  at  most  a  factor  roughly  equal  to  the  probability  that 
failure  ultimately  occurs.  The  estimate  will  in  general 
be  biased  when  all  the  path  data  are  used  in  a  non-Markov 
context. 

„/  •  The  most  difficult  step  in  defining  a  useful  operational 

Markov  model  is  expected  to  be  determining  a  satisfactory 
set  of  states  (  "state  space  ").  A  set  of  reasonable  re¬ 
quirements  for  the  state  space  are: 

-  States  should  be  mutually  exclusive  and  exhaustive. 

-  Each  state  should  be  operationally  observable. 

-  Each  state  should  be  of  operational  significance. 

-  States  should  be  of  reasonable  generality,  though  not 
necessarily  so  general  as  the  traditional  state  sequence 
of  Opportunity  +  Detect  4-  Classify  4-  Attack  4-  Kill. 

-  States  should  be  defined  so  that  the  effectiveness 
measure(s)  can  be  calculated  by  probability  statements 
about  state  occupancy  or  by  mean  time-to-occupy  state¬ 
ments.  This  often  can  be  accomplished  by  defining  success 
and  failure  states  as  absorbing  states. 

-  The  statistical  requirement  that  knowledge  only  of  the 
current  state  should  determine  the  transition  probabilities 
for  the  next  step  should  be  reasonably  well  met. 

•  Effectiveness  (and  performance)  measures  can  be  defined  at 
two  levels  in  a  Markov  model.  Lower-level  measures  often 
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relate  to  the  performance  of  individual  subsystems  or  ASW 
functions  such  as  detection  and  classification.  These 
measures,  which  are  quantities  such  as  Prob(Classify/Detect) 
and  the  mean  time  to  classify,  can  be  associated  with  the 
microstructure  of  the  model;  that  is,  with  transition 
probabilities.  Overall  effectiveness  measures  that  depend 
on  the  entire  set  of  model  parameters  are  at  a  different 
level. 

-  Degrading  factors  such  as  false  contacts  can  be  explicitly 
included  to  permit  systematic  study  of  the  relationship 
between  the  degradation  parameters  and  the  effectiveness 
measures. 

-  Many  calculations  of  interest  can  be  routinely  made;  in 
particular,  the  dynamic  response  can  be  calculated  when 
appropriate  and  sensitivity  analysis  can  be  routinely 
performed. 

-  Many  kinds  of  Fleet  operational  data,  both  static  and 
dynamic,  can  in  theory  be  used  to  systematically  estimate 
the  parameters  of  a  Markov  model.  One  type  of  useful 
static  data  (transition  data)  is  now  used  to  estimate  the 
the  conditional  probability  parameters  in  conditional 
probability  models.  Other  types  of  potentially  useful 
data  are  rarely  if  ever  used  in  effectiveness  modeling; 
an  example  is  average  time-in-state  data  that  may  be 
considered  dynamic  data. 

-  Standard  methods  are  available  for  many  parameter  estima¬ 
tion  problems  in  Markov  modeling.  Other  methods  were  in¬ 
vestigated  and  evaluated  during  the  current  research. 

Many  parameter  estimation  problems  lead  to  least  squares 
or  quadratic  programming  problems.  When  a  satisfactory 
set  of  states  has  been  found  for  a  Markov  model,  parameter 
estimation  is  not  expected  to  be  a  significant  problem. 

-  Operational  Markov  models  may  be  used  to  obtain  improved 
estimates  of  certain  of  the  traditional  effectiveness 
measures. 

Selecting  a  suitable  set  of  states  for  an  operational  Markov 
model  in  ASW  will  often  require  the  aggregation  of  states 
and/or  the  addition  of  auxiliary  states  to  make  the  Markovian 
assumption  valid.  It  was  found  during  the  research  that 
auxiliary  states  can  often  improve  the  fit,  and  specific 
methods  are  given  for  introducing  them.  Aggregation  of 
states  by  systematic  methods  is  more  difficult;  one  method 


sometimes  useful  is  least  squares  using  Prob( state)  as  a 
function  of  time  as  input.  As  a  general  method,  least 
squares  appear  to  offer  several  uses  in  the  structuring  of 
the  state  space  and  in  estimating  parameters.  The  general 
use  seems  to  result  from  the  linearity  of  many  of  the  sets 
of  equations  associated  with  Markov  theory.  Quadratic  pro¬ 
grams  are  needed  when  the  transition  probabilities  esti¬ 
mated  from  least  squares  are  infeasible. 

•  Markov  models  may  be  useful  in  comparing  dissimilar  models 
and  in  validating  simulation  models.  The  comparison  (or 
validation)  can  be  accomplished  by  determining  a  suitable 
common  set  of  states  for  the  two  models,  estimating  the 
probability  of  occupying  each  state  for  all  relevant  in¬ 
stants  of  time  for  each  model  separately,  and  comparing 
the  two  sets  of  Prob( state)  as  a  function  of  time  curves. 

•  Current  analysis  reports  from  operational  exercises  contain 
inadequate  data  for  Markov  modeling.  Only  minor  changes  in 
analysis  procedures  should  be  required  to  collect  data  for 
a  Markov  model  with  a  moderate  number  of  states,  however. 

•  A  Markov  model  may  be  useful  in  resolving  some  experimental 
design  problems  encountered  in  Fleet  exercise  planning.  One 
simple  situation  in  the  study  was  analyzed  by  using  an  opti¬ 
mization  method  that  incorporated  the  cost  of  experiments. 
The  optimum  number  of  samples  of  each  available  type  were 
determined  analytically  on  the  assumption  that  the  objective 
would  minimize  the  variance  of  the  Prob( success)  estimator 
subject  to  the  cost  constraints. 


C.  Research  Activities 

The  findings  in  Section  B  were  arrived  at  by  considering  the  combined 
research  activities  from  both  phases  of  the  project.  An  overview  of  the 
individual  sections  of  this  report  conveniently  summarizes  these  activities. 

A  summary  of  the  work  done  on  queueing  models  during  the  first  phase 
of  the  project  is  given  in  Section  II.  This  section  also  introduces  some 
results  from  the  analytical  queueing  models  that  obtained  after  the  pub¬ 
lication  of  the  interim  report.1 
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In  Section  III,  conditional  probability  models  are  defined,  dis¬ 
cussed,  and  shown  to  be  a  special  form  of  Markov  model.  Parameter  esti¬ 
mation  for  this  model  is  briefly  discussed,  and  some  intuitive  attempts 
are  made  to  introduce  false  contacts  by  modifying  the  estimation  formula. 
The  modified  formulas  prove  to  be  incorrect,  however,  and  the  correct 
formula  (derived  from  Markov  theory)  is  given.  Although  the  limitations 
of  conditional  probability  models  are  enumerated,  several  arguments  are 
given  to  justify  their  use.  Finally,  the  suggestion  is  given  that  many 
of  the  limitations  on  conditional  probability  models  can  be  removed  by 
generalizing  to  a  Markov  model. 

Section  IV  opens  with  a  brief  general  discussion  of  useful  properties 
of  Markov  models  in  ASW  effectiveness  analysis.  Four  measures  of  effec¬ 
tiveness  of  rather  general  use,  but  readily  definable  and  calculable  in 
a  Markov  model,  are  then  given.  Also  discussed  is  model  formulation  that 
includes  defining  a  set  of  states  so  that  the  measures  may  be  conveniently 
calculated.  State  space  definition,  asserted  to  be  the  most  difficult 
step  in  model  formulation,  and  rules  of  thumb  for  formulating  a  set  of 
states  are  given.  Parameter  estimation  in  a  Markov  model  applied  to  an 
ASW  situation  is  briefly  discussed.  Finally,  the  possibility  is  mentioned 
of  using  operational  data  not  currently  used  for  modeling. 

Basic  Markov  chain  theory  and  formulas  are  given  in  Section  V.  This 
material  is  needed  for  either  analytical  or  operational  modeling;  for  the 
latter,  it  is  needed  both  for  calculating  effectiveness  measures  and  as  a 
basis  for  several  parameter  estimation  methods.  Algebraic  formulas  for 
all  the  measures  are  included.  Common- sense  interpretations  of  the  ab¬ 
sorption  probability  formulas  and  their  elements  are  given  where  appro¬ 
priate  to  aid  the  understanding. 

In  Section  VI,  parameter  estimation  for  a  Markov  model  is  discussed 
at  length.  We  point  out  the  usefulness  of  factoring  the  estimation  problem 
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into  static  and  dynamic  parts.  Parameters  for  the  static  part,  which 
correspond  to  a  conditional  probability  model  (or  the  imbedded  chain  in 
Markov  theory),'  may  be  estimated  from  transition  data  only.  The  dynamic 
parameters  can  then  be  separately  estimated  from  mean  time- in-state  opera¬ 
tional  data. 

Methods  for  estimating  transition  probabilities  are  then  given  for 
various  assumptions  about  the  data  available.  The  input  data  are  assumed 
to  be  transition  data,  mean  time- in-state  data,  mean  first-passage  time 
data,  data  on  the  first  and  second  moments  of  first  passage  times,  or 
probability  of  state  as  a  function  of  time  data.  The  need  for  least 
squares  and  quadratic  programming  are  shown  in  several  instances,  but 
these  are  discussed  in  more  detail  later  in  the  section. 

Holding  (or  waiting)  times  in  a  state,  which  are  a  function  of  the 

self- transition  probabilities  (p  )  in  a  Markov  chain,  are  the  subject 

li 

of  the  remainder  of  the  section.  The  p  estimator  is  first  shown  to  be 

n 

a  simple  function  of  the  observed  mean  time  in  state  i  in  a  Markov  chain. 
Then,  various  ways  to  better  approximate  the  holding  time  behavior  by  en¬ 
larging  the  chain  through  the  addition  of  auxiliary  states  are  suggested 
and  examples  are  given.  The  section  concludes  with  some  chi-squared 
goodness-of-f it  tests  that  may  be  used  to  test  the  reasonableness  of  the 
Markov  assumption. 

Section  VII  deals  with  the  statistical  problem  of  reducing  the 
variance  of  the  four  effectiveness  measures  by  using  a  Markov  model. 
Improvement  is  measured  relative  to  the  estimates  obtainable  directly 
from  the  data.  Certain  mathematical  identities  (proved  as  theorems  in 
an  appendix)  are  the  key  tool  for  discussing  this  important  problem  area. 
Several  implications  of  the  theorems  conclude  the  section. 

Anticipating  that  most  real-world  ASW  problems  will  not  have  a  Markov 
structure  (at  least  when  a  natural  set  of  states  is  used),  we  need  to  know 
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how  well  Markov  methods  work  on  non-Markov  problems.  Therefore,  in  Sec¬ 
tion  VIII,  a  dynamic  non-Markov  model — conveniently  described  by  a 
Probability-State-Time  (PST)  diagram — is  introduced,  motivated,  and  re¬ 
lated  to  a  Markov  model.  A  PST  diagram  is  used  in  conjunction  with  a 
dynamic  (Markov)  version  of  the  WSE  model  in  Section  IX.  Numerical  ex¬ 
perimentation  uses  data  from  the  PST-defined  model,  and  the  resulting 
dynamic  behavior  given  is  compared  with  the  dynamics  of  the  actual  process. 

Various  methods  are  used  to  estimate  parameters,  and  auxiliary  states 
are  sometimes  added  along  the  lines  of  the  suggestions  in  Section  VI. 
Although  the  addition  of  auxiliary  states  improved  the  dynamic  fit,  all 
Markov  chain  models  fit  rather  poorly.  Therefore,  two  more  sophisticated 
Markov  models  were  tried.  The  first,  a  semi-Markov  model,  resulted  in  a 
substantial  improvement.  The  last  attempt  used  time  itself  in  the  state 
definitions:  The  result  was  an, essentially  perfect  dynamic  fit  at  the 
cost  of  a  greatly  increased  number  of  states. 

In  Section  X,  three  false-contact  models  were  defined  by  adding 
states  for  the  prosecution  of  false  contacts  to  the  traditional  states — 
search,  opportunity,  detect,  classify,  and  kill.  Markov  formulas  are 
given  for  the  Prob(Success)  effectiveness  measure;  the  form  is  a  product 
of  conditional  probabilities  divided  by  a  false  contact  correction  factor. 
For  the  second  of  these  models,  the  influence  of  false  contacts  on  effec¬ 
tiveness  takes  the  form  of  a  multiplicative  correction  factor  to  apply  to 
the  traditional  product  of  conditional  probabilities.  The  third  and  most 
complex  false  contact  model  was  defined  by  a  cartesian  product  method;  it 
has  27  states  and  includes  false  attacks  and  earlier  stages  of  false- 
contact  prosecution. 

Further  numerical  expectation,  with  a  PST  diagram  for  the  first  false 
contact  model  in  Section  X — is  performed  in  Section  XI.  Five  methods  were 
used  to  estimate  parameters  with  the  original  state  space.  Three  of  these 
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methods  were  semi-Markov  and  the  other  two  were  Markov  chains;  none 
worked  well.  Monte  Carlo  experiments  were  also  performed  to  study 
sampling  problems. 

After  the  earlier  parameter  estimation  attempts  failed  with  the 
original  state  space,  some  methods  were  tried  with  auxiliary  states. 

Some  improvement  resulted  from  the  most  elaborate  of  these  models,  but 
the  fit  remained  unsatisfactory.  By  the  end  of  the  experimentation,  the 
reasons  for  the  many  failures  became  apparent. 

A  new  topic,  Markov  models  and  Monte  Carlo  simulation,  is  the  sub¬ 
ject  of  Section  XII.  A  single  Monte  Carlo  model  including  a  submarine, 
an  aircraft  carrier,  and  an  acoustic  decoy  was  selected  for  study  with 
the  use  of  the  mean  survival  time  measure.  The  idea  was  to  use  the  simu¬ 
lation  to  generate  data  for  estimating  the  parameters  of  a  Markov  chain 
model.  A  simple,  natural  set  of  states  proved  to  be  satisfactory  for 
this  simulation.  Parameters  were  determined  in  two  different  ways  and 
led  to  almost  identical  estimates.  Probability  of  state  as  a  function 
of  time  comparisons  were  made  between  the  approximating  Markov  chain  and 
the  simulation  model's  direct  output  for  assessing  the  validity  of  the 
approximation.  Finally,  some  basic  sensitivity  analyses  were  performed 
by  using  the  fitted  parameters,  and  an  estimate  was  made  of  the  increase 
in  survival  time  as  a  result  of  an  increase  in  the  primary  decoy  parameter 
(called  "capture  time") . 

Section  XIII  deals  with  two  questions:  how  to  combine  different 
estimates  of  the  same  parameter  in  an  optimal  way,  and  how  to  use  a  Markov 
model  in  exercise  design.  Both  of  these  questions,  which  are  treated  in 
a  very  limited  manner,  entail  the  use  of  estimates  of  variance.  For 
question  one,  formulas  are  given  for  combining  two  or  three  independent 
estimates  of  an  unknown  constant,  and  the  formulas  are  applied  to  the 
transition  probabilities  in  a  Markov  chain.  To  answer  the  second  question, 
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an  example  of  a  simple  three-state  Markov  chain  model  is  assumed;  a  fur¬ 
ther  assumption  is  that  the  number  of  samples  at  each  of  the  two  transient 
states  can  be  freely  chosen.  Formulas  are  then  derived  for  the  optimum 
number  of  samples  under  a  cost  constraint,  and  a  table  is  provided  to 
illustrate  the  optimal  sample  sizes  for  various  costs. 


D.  Possible  Operational  Applications 


Possible  operational  applications  are  discussed  in  Section  XIV,  the 
final  section.  Many  forms  of  false  contact  models  are  possible  using 
Markov  concepts.  The  choice  of  a  state  depends  on  the  application  and 
the  problem  features  the  analyst  wishes  to  distinguish.  Markov  models 
also  may  be  helpful  in  studying  the  difficult  subject  of  countermeasure 
effectiveness  because  states  can  be  defined  in  terms  of  information. 
States  and  Markov  models  are  then  discussed  in  relation  to  the  UPTIDE 
series  of  exercises,  and  suggestions  are  made  for  analysis  using  some  of 
the  concepts  from  this  report.  Other  possible  applications  include 
modeling:  range  from  a  submarine  and  an  HVU,  electromagnetic  radiations 

from  an  aircraft  carrier,  submarine  exposure,  and  the  submarine's  classi¬ 
fication  process. 

Two  recommendations  conclude  the  body  of  the  report: 

(1)  A  research  project  in  an  area  of  current  interest  to  the 
Navy  should  be  undertaken  to  apply  the  dynamic  Markov 
modeling  methods  developed  during  the  study.  The  model 
should  be  developed  for  class  of  tactical  problems  where 
wholly  satisfactory  measures  of  effectiveness  have  not  yet 
been  developed  because  of  the  dynamic  nature  of  the  prob¬ 
lem,  the  complexity  of  the  problem,  or  the  difficulty  of 
relating  lower  level  performance  measures  to  the  overall 
effectiveness  measures.  The  Naval  Analysis  Programs  Office 
of  the  Office  of  Naval  Research  could  sponsor  such  research 
as  a  continuation  of  the  Dynamic  Analysis  study. 

(2)  The  technique  of  using  a  Markov  model  to  compare  dissimilar 
models  or  to  assist  in  validating  a  simulation  model  should 
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be  suggested  to  Navy  analysts  concerned  with  simulation 
validation  and  effectiveness  modeling. 

Six  appendices  follow  the  body  of  the  report.  Appendix  A  discusses 
ASW  modeling  which  is  conceptually  related  to  the  report,  and  compares 
the  other  approaches  with  those  in  the  interim  report  and  the  present 
report.  Appendix  B  treats  a  special  topic  which  relates  to  the  analytical 
Queueing  Models.  Appendix  C  gives  Mason's  rule  for  determining  trans¬ 
missions  in  a  flowgraph;  the  rule  was  applied  to  obtain  the  measures  of 
effectiveness  for  the  two  of  the  false  contact  models.  Appendix  D  dis¬ 
cusses  least  squares  and  quadratic  programming  that  were  needed  for 
parameter  estimation.  Appendix  E  has  proofs  of  the  three  important  iden¬ 
tities  relating  to  parameter  estimation.  Finally,  Appendix  F  amplifies 
the  earlier  discussion  of  state  selection. 
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II  SUMMARY  OF  QUEUEING  MODELS  OF  ASW  UNITS 


A.  Summary  of  Work  During  the  First  Phase 

During  the  first  year  of  the  contract,  the  emphasis  was  on  modeling 
ASW  operations  in  terms  of  queueing  theory.1  The  objective  was  improved 
representation  of  ASW  dynamics  that  would  lead  to  more  realistic  measures 
of  effectiveness  than  those  currently  available  with  traditional  formulas 
consisting  of  products  of  success  probabilities  (the  WSE  model,  for  ex¬ 
ample).  Because  "congestion"  from  many  simultaneous  contacts  causes  some 
degree  of  delay  in  ASW  system  processing,  the  suggestion  to  attempt 
queueing  models  in  ASW  analysis  arose. 

After  the  literature  on  queueing  theory  and  its  applications  was  re¬ 
viewed,  the  conclusion  was  that  a  direct  application  of  readily  available 
results  on  single  queue/multiple  server  systems  is  infeasible.  ASW  situa¬ 
tions  simply  do  not  exhibit  properties  of  a  single  "waiting  line"  in  any 
sense.  What  was  needed  to  relate  to  ASW  appeared  to  a  degree  in  articles 
on  networks  of  queues,  such  as  in  models  of  time-sharing  and  other  com¬ 
puting  systems.  Again,  however,  no  direct  parallel  was  found  because 
contacts  could  not  be  justifiably  regarded  as  passing  through  a  series  of 
ASW  process  stages  without  proper  consideration  of  strong  interaction  be¬ 
tween  the  stages.  Therefore,  the  decision  was  made  to  construct  a  new 
set  of  congestion  models  that  would  exhibit  queueing-like  characteristics, 
be  amenable  to  analysis  by  techniques  in  queueing  theory,  and  relate 
better  to  the  ASW  situations  considered.  This  resulted  in  queueing 
analysis  techniques  found  for  solution  of  the  models  created;  the  key 
technique  was  the  method  of  imbedded  Markov  chains.  Additional  techniques 
found  of  great  value  in  computations  with  the  imbedded  Markov  models  were: 
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•  Laplace  transform  solutions. 

•  Flowgraph  techniques  to  organize  computations  efficiently. 

•  Matrix  flowgraphs  to  aid  in  model  formulation. 

•  Successive  over  relaxation  (SOR)  method  for  solving  large 
systems  of  linear  differential  equations  associated  with 
Markov  processes. 

•  Pade  approximants  to  e^  for  speeding  up  numerical  integra¬ 
tion  with  Markov  models. 

Other  methods  useful  in  queueing  analysis  were  not  adaptable  to  ASW  models 
considered  on  this  project.2  These  methods  include 

•  Contour  integration  of  Laplace  transforms  for  waiting  time 
distribution  in  tandem  queues. 

•  Integral  equations  for  waiting  time  distributions. 

•  Integro-dif ferential  equations  for  transient  waiting  time 
distributions . 

Although  the  queueing  literature  reports  very  powerful  results  with  the 
above  techniques,  these  methods  are  mainly  for  a  single  queue;  no  clear 
way  appears  as  feasible  as  the  Markov  techniques  for  generalizing  their 
applicability  to  more  complex  congestion  models. 

The  work  on  queueing  models  was  divided  into  two  areas:  at  the 
single  ASW  unit  level  and  at  the  force  level.  This  two-pronged  approach 
decision  was  made  on  the  basis  that  congestion  in  ASW  operations  probably 
originates  in  processing  delays  of  contact  information  on  individual  ASW 
screen  or  search  units,  but  that  the  overall  effect  is  on  the  force  level 
with  regard  to  decisions  to  prosecute,  maneuver  for  improved  contact  data, 
or  evade  for  safety.  Consequently,  work  was  started  on  the  following  two 
separate  but  closely  related  models. 

•  A  model  for  analysis  of  congestion  effects  in  an  ASW  unit 
providing  distributions  for  its  periods  of  saturated  and 
unsaturated  detection  capability. 
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•  A  model  for  computing  the  probability  of  detecting  a  target 
by  a  typical  ASW  force  configuration  of  units  with  known 
saturated  and  unsaturated  period  distributions  and  detection 
versus  range  characteristics.  A  summary  discussion  of  these 
models  is  presented  in  the  following  two  subsections. 

1.  ASW  Unit  Model 

For  the  unit  ASW  model  it  was  assumed  that  as  a  contact  passes 
through  the  system,  it  occupies  a  unit  of  sonar  service  while  in  the  first 
"server" — i.e.,  while  undergoing  detection — and  continues  to  occupy  that 
unit  of  sonar  service  even  when  in  any  of  the  three  queues  following  the 
first  server.  Similarly,  the  Combat  Information  Center  (CIC)  resource 
is  assumed  to  be  occupied  at  the  rate  of  one  unit  per  contact  from  the 
moment  of  entering  "localization"  and  up  to  leaving  the  system  and  so  on. 

The  unit  model  assumed  that  each  ASW  function  serves  a  limited 
number  of  multiple  contacts  simultaneously.  The  unit  model  representa¬ 
tion  weighs  all  contacts  uniformly  for  the  load  each  places  on  a  server, 
or  equivalently,  for  the  difficulty  of  service. 

The  command  and  control  and  classification  functions  were 
omitted  from  the  initial  model.1  Similarly,  certain  other  features  of 
an  ASW  engagement  have  not  been  modeled.  Some  of  these  are:  hand-off 
of  functions  from  other  ASW  units,  lost  contact  processes,  and  unsuccess¬ 
ful  kill  attempts  and  reattack  process. 

Based  on  the  above  considerations,  the  initial  multiple  queueing 
situation  of  Figure  2.1  was  selected  for  study.  The  meaning  of  Figure 
2.1  follows. 

Acoustic  stimuli,  if  they  are  locally  generated  as  they  become 
available  for  detection,  await  the  availability  of  the  detection  server 
D  in  a  fictitious  queue  Q^.  This  queue  is  created  to  measure  the  statis¬ 
tics  of  missed  and  delayed  detection  caused  by  preoccupation  of  the 
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FIGURE  2.1  PROTOTYPE  QUEUING  REPRESENTATION  OF  AN  ASW  SYSTEM 

detection  process.  As  soon  as  an  element  of  the  detection  server  is 

available  (four  degrees  of  detection  server  occupancy  are  used  here  for 

specificity),  the  stimulus  progresses  "through'1  the  detection  phase 

according  to  prescribed  detection  time  statistics.  From  this  point  the 

stimulus  is  a  contact.  Contacts  not  in  process  of  localization  are 

represented  as  "waiting"  in  a  queue  Q  in  the  same  manner  as  the  contacts 

L 

progressed  from  a  queue  awaiting  detection  to  a  queue  awaiting  localiza¬ 
tion.  So,  the  contacts  progress  from  the  latter  queue,  through  the 
localization  process,  through  a  queue  awaiting  the  attack  function,  and 
so  on,  through  service  by  the  kill  function. 

The  number  of  parallel  service  units  at  each  stage  was  selected 
for  development  purposes  to  be  just  enough  so  that  queueing  is  possible 
at  the  following  stage.  That  is,  in  view  of  service  at  each  stage  being 
"tied  up"  until  a  contact  leaves  the  system,  at  least  one  less  unit  of 
service  is  necessary  at  each  successive  stage.  In  other  words,  the  server 
capacities  were  chosen  to  be  simple  enough  for  a  first  study,  but  compli¬ 
cated  enough  to  allow  queues  to  form  in  all  possible  stages. 

An  alternative  schematic  representation  of  the  situation  is 
presented  in  Figure  2.2.  This  figure  shows  the  idea  of  the  presence  of 
a  contact  being  felt  in  several  places  at  once,  in  that  a  contact 


18 


CONTACTS 


STIMULI 


SIMULTANEOUS 


FIGURE  2.2  ALTERNATIVE  REPRESENTATION  FOR  FIGURE  2.1 
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progressively  uses  the  resource  of  sonar  (detection),  then  sonar,  CIC 
(localization),  and  so  on,  until  it  finally  exits  from  the  system. 

As  a  start  it  was  assumed  that  all  parallel  service  units  at 
each  stage  have  the  same  exponential  service  distribution,  with  parameters 
as  in  Figure  2.1.  It  must  be  remembered  that  a  completion  of  a  service 
does  not  make  a  given  service  unit  available  for  other  contacts;  such  a 
completion  merely  tells  when  a  transition  of  a  contact  to  the  next  state 
occurs  (either  into  the  next  server  or  the  queue  ahead  of  .it).  The  in¬ 
put  distribution  for  all  true  and  false  contacts  is  also  assumed  to  be 
exponential  with  parameter  \.  (Both  the  input  and  service  distributions 
can  be  made  more  general  by  using  a  network  of  parallel  and  series  ex¬ 
ponential  holding  times.) 

Various  attributes  of  factors  associated  with  queueing  systems 
in  general  were  considered  in  structuring  a  suitable  model  for  evaluating 
ASW  system  performance.  The  primary  statistical  distributions  were  judged 
to  be : 

•  Unsaturated  time  distribution  (for  entire  system). 

•  Saturated  time  distribution  (for  entire  system). 

•  Waiting  time  distribution  (in  first  queue,  i.e.,  ahead 
of  detection) . 

•  Distribution  of  time  spent  in  entire  system. 

Other  significant  distributions  are: 

•  Queue  lengths  at  the  various  stages. 

•  The  number  of  contacts  exiting  any  stage  in  a  given 
time . 

The  primary  statistical  distributions  are  needed  in  the  force  effective¬ 
ness  model.1  The  others  may  be  useful  for  evaluating  the  effects  of 
physical  storage  capacity  or  other  limitations  at  various  stages  of  the 
ASW  system  and  for  determining  the  nature  of  the  input  distributions  to 
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other  ASW  platform  queueing  models  coupled  to  a  given  one  through  a 
command  and  control  hand-off  system. 

The  above  statistical  distributions  can  be  used  to  calculate 
the  following  possible  measures  of  effectiveness  of  an  ASW  unit: 

•  Probability  that  a  contact  is  lost. 

•  Expected  number  of  contacts  lost  while  in  the  system, 
over  a  given  period  of  time. 

•  Probability  of  missing  a  true  contact  (failure  to  de¬ 
tect)  for  a  given  level  of  contact  activity  (false  and 
true) . 

•  Expected  time  to  detect. 

•  Expected  time  from  detection  to  kill. 

If  the  classification  function  were  added  to  the  model,  the 
following  additional  measures  would  be  obtainable: 

•  Probability  of  losing  a  contact  before  classifying. 

•  Expected  time  to  classify. 

•  Probability  that  a  false  contact  is  pursued  all  the 
way  through  to  the  kill  function. 

Computation  was  aimed  only  at  numerical  calculation  of  the 
statistical  distributions  as  these  are  the  most  difficult  to  obtain. 

The  above  measures  of  effectiveness  were  not  considered  in  the  current 
effort  because  fleet-operational  data  were  unavailable  to  render  them 
meaningful.  Graphs  comparing  analytical  solutions  and  simulation  re¬ 
sults  for  some  of  the  computations  that  were  performed  are  presented  in 
Section  II-B. 


2.  ASW  Force  Effectiveness  Model 

The  force  level  queueing  model  developed  during  the  first  phase 
of  this  study  considered  an  ASW  screening  situation  in  which  ASW  units 
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maneuver  within  a  protected  area  surrounding  a  task  force.  The  individual 
ASW  units  were  assumed  to  experience  congestion  from  the  abundance  of 
contacts  according  to  the  unit  queueing  model  previously  described.  In 
the  numerical  example  worked  out,  only  exponential  approximations  to  the 
distributions  of  the  unit  model  were  used. 

The  force  model  was  based  on  the  idea  that  units  could  be  com¬ 
bined  in  a  single  detection  device  whose  statistics  for  "windows"  of 
possible  target  capture  result  from  joint  probability  of  detection  by 
any  one  of  the  ASW  units  within  range  of  a  target.  Kinematics  of  both 
targets  and  ASW  units  were  included  along  with  a  simple  range-law  for 
probability  of  detection  given  opportunity.  The  example  used  for  compu¬ 
tations  is  described  by  Figure  2.3  that  shows  initial  positions  and 
velocity  vectors  of  the  ASW  units  participating.  A  lengthy  series  of 
Calculations  was  performed  to  obtain  probabilities  of  a  target  ever  being 
detected  prior  to  its  penetrating  the  protected  zone,  given  that  the  tar¬ 
get  approaches  the  force  from  all  possible  (quantized)  positions  of 
initial  appearance,  as  indicated  in  Figure  2,4  (all  probabilities  have 
been  multiplied  by  100  in  the  figure). 

The  assumption  in  combining  ASW  units  into  a  force  was  that 
each  unit  individually  experiences  alternating  periods  of  being  saturated 
and  not  saturated  by  contacts  it  is  processing,  as  determined  by  the  unit 
model.  In  addition,  a  waiting  time  distribution  accounted  for  the  first 
period  of  the  series  of  busy  and  idle  periods  of  any  ASW  unit — with  re¬ 
gard  to  the  time  arrival  of  a  specific  contact — because  the  given  contact 
could  arrive  during  either  a  busy  or  idle  time.  The  probability  that  a 
target  is  ever  detected  in  time  for  effective  counteraction  was  expressed 
in  terms  of  probabilities  of  detection  by  each  of  the  sensors  during  each 
of  their  unsaturated  periods  up  until  the  time  that  the  target  penetrates 
the  protected  zone. 
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FIGURE  2.4  ILLUSTRATIVE  FORCE  EFFECTIVENESS  MODEL  RESULTS 
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B. 


Further  Computations  with  the  Single  Unit  Model 


The  most  complex  of  all  flowgraphs  introduced  in  the  previous  year’s 
work  on  a  queueing  model  of  a  single  ASW  unit  was  for  "time  in  the  sys¬ 
tem,"  i.e.,  the  time  from  the  moment  a  contact  arrives  at  the  detection 
periphery  to  the  time  that  it  is  finally  killed.  Since  only  true  targets 
were  considered,  all  contacts  were  assumed  to  be  eventually  localized, 
attacked,  and  then  killed.  The  same  methodology  would  apply  equally 
well,  however,  to  a  more  general  semi-Markov  model  with  classification, 
false  contacts,  and  possible  loss  of  contacts. 

Because  of  the  large  number  of  states  in  the  time-in-system  flow- 
graph  (see  Ref.  1,  p.  III-39  and  Figures  III-19  to  III-22),  actual  compu¬ 
tations  of  distributions  for  this  case  were  not  completed  during  the 
first  year's  work.  Still  more  efficient  numerical  techniques  had  to  be 
set  up  early  in  the  current  year  before  results  were  obtainable.  In 
addition  to  the  continued  analytical  work,  a  Monte  Carlo  simulation  model 
was  prepared  and  run  in  parallel  with  the  analytical  model.  Both  ap¬ 
proaches  led  to  almost  identical  plots  for  all  distributions,  i.e.,  con¬ 
tact  waiting  time,  ASW  system  saturated  and  unsaturated  time,  and  time 
to  process  a  specified  contact  from  opportunity  to  kill.  The  results 
for  these  distributions  and  the  simulation  data  will  be  described  further 
below. 


1.  Special  Numerical  Integration  Method  for  Solving  Large  Flowgraphs 


A  special  integration  technique,  SOR,  was  used  to  handle  the 

* 

large  size  of  the  Markov  model  for  contact  time  in  the  system.  The 
advantage  of  using  this  method  and  an  explanation  of  it  are  laid  out  in 


We  are  indebted  to  Dr.  Samuel  Schecter  who  recommended  the  methods  used 
here . 
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Appendix  B  of  Ref.  1.  Steps  were  taken  to  reduce  integration  time  of 
this  large  system  of  equations  to  as  low  a  value  as  possible  per  inte¬ 
gration  step.  Some  convergence  difficulties  were  encountered  for  the 
over-relaxation  iterations,  as  the  method  is  quite  critical  with  regard 
to  choice  of  the  over-relaxation  parameter  u).  A  detailed  discussion  of 
the  problems  associated  with  the  selection  of  a)  is  given  in  Appendix  B 
of  this  report. 

The  analytically  computed  probability  distribution  for  contact 

* 

time  in  the  system  is  given  in  Figure  2.5  along  with  the  simulation  data. 
Almost  identical  results  were  obtained  by  the  two  methods  up  to  t  =  0.6 
hours,  but  the  analytical  integration  was  not  carried  out  any  further  be¬ 
cause  of  the  relatively  high  computing  cost  compared  to  simulation  cost. 

2.  Results  of  a  Simulation  Approach 

Although  analytical  modeling  was  the  primary  concern  of  the 
first  phase  of  the  current  project,  it  was  decided  to  compare  results 
obtained  so  far  with  data  collected  from  a  direct  simulation  of  the  single 
ASW  unit  model.  Use  was  made  of  the  General  Activity  Simulation  Program 
(GASP).  The  results  obtained  from  simulation  were  essentially  identical 
to  the  analytical  results.  Figures  2.6(a),  (b),  and  (c)  are  copies  of 
the  analytical  results  (Figure  III-26  of  Ref.  1)  with  the  simulation  re¬ 
sults  superimposed.  Similarly,  Figure  2.7  and  Figure  2.2  demonstrate 
the  comparison  of  steady-state  and  time- in- system  results. 


Each  integration  "step"  here  is  not  the  usual  "infinitesimal"  one,  how¬ 
ever,  as  we  are  using  Pade  approximants  of  e^^  accurate  up  to 
l|AAt||  ~  1  or  2  as  opposed  to  0.001,  or  something  of  that  order,  in  com¬ 
mon  integration.  Hence,  only  a  few  "steps"  are  used  in  total — by  our 
method — to  achieve  enough  data  points  to  plot  curves  for  the  desired 
distributions. 
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FIGURE  2.6(c)  PROBABILITY  DISTRIBUTIONS  FOR  WAITING  TIME 
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STEADY  STATE  PROBABILITY  (  P) 


The  fact  that  simulation  verifies  the  results  of  the  analytical 
work  is  rewarding  because  rather  complex  analytical  modeling  and  computa¬ 
tion  were  performed.  The  slight  differences  between  some  of  the  simula¬ 
tion  and  analytical  curves  probably  result  from  statistical  fluctuations. 
(The  simulation  was  run  for  an  equivalent  of  up  to  10,000  hr  of  model 
time,  thus  processing  a  total  of  about  35,000  contacts  for  an  arrival 
rate  of  3. 5/hr. ) 

A  disappointing  feature  of  the  above  comparison  (in  terms  of 
the  desirability  of  an  analytical  approach)  is  that  the  simulation 
approach  took  a  great  deal  less  computer  effort  to  obtain  some  of  the 
results,  even  with  highly  dimensional  Markov  models.  Thus,  further 
analytical  work  in  the  direction  of  more  realistic  models — including 
classification,  nonexponential  distributions  for  contact  arrivals,  and 
the  like — during  the  current  year  had  to  be  restricted  to  Markov  and 
semi-Markov  models  of  modest  dimensions,  having  in  the  order  of  3  to  28 
states. 

C.  Proposed  Unit  Model  Extensions  to  Represent  Classification 

The  process  of  classifying  contacts  was  omitted  from  the  queueing 
models  of  the  first  year  of  the  current  project  so  that  complexity  of 
initial  models  could  be  reduced  and  analytical  approaches  could  be  tested 
on  more  tractable  computational  problems.  This  section  describes  how 
classification  may  be  included  within  the  same  analytical  framework. 

To  include  classification  in  the  queueing  models,  the  dimensionality 
of  an  associated  Markov  process  imbedded  in  the  queueing  process  has  to 
be  increased.  Classification  is  basically  a  decision  process  whose  func¬ 
tion  is  to  separate  true  from  false  contacts;  therefore,  at  least  these 
two  categories  of  contacts  must  be  considered.  In  practice,  several 
categories  are  used,  but  model  construction  was  begun  with  the  categories 
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possible  sub,  nonsub,  and  certain  sub.  Furthermore,  classification  is 
not  always  perfect;  hence,  there  will  be  actually  six  significant  cate¬ 
gories  of  contacts  as  shown  in  Table  2.1.  The  false  nonsub  category  acts 

Table  2.1 

SIGNIFICANT  CATEGORIES  OF  CONTACT 


Contacts 

Classified 

True 

Possible  sub 

Certain  sub 

Nonsub 

False 

Possible  sub 

Certain  sub 

Nonsub 

as  a  "sink";  that  is,  any  contacts  falling  into  it  are  assumed  not  to  be 
picked  up  again  as  possible  subs.  All  contacts  start  out  as  possible 
subs,  by  the  definition  of  contact.  It  is  presumed  that  true  and  false 
contacts  might  obtain  different  transition  rates  among  the  three  classi¬ 
fication  categories.  Also,  all  transition  rates  would  probably  vary 
with  the  contact's  position  in  the  queueing  system  and  with  the  degree 
of  congestion  of  the  system. 

Two  main  proposed  approaches  to  specific  formulation  of  a  single 
ASW  unit  queueing  model  have  evolved  during  the  current  project. 

Model  1 — Classification  is  assumed  to  be  performed  "instantly"  just 
before  any  previously  modeled  transition  of  the  Markov  model  (except  for 
an  arrival),  i.e.,  at  the  completion  of  a  detection  service,  a  localiza¬ 
tion  service,  and  the  like.  This  approach  is  relatively  easy  to  imple¬ 
ment  because  it  merely  modifies  the  manner  in  which  transitions  take 
place  in  the  Markov  models  developed  to  date  and  extends  them  to  include 
the  extra  categories  of  contacts  mentioned  above.  Thus,  just  at  the 
completion  of  detection  of  a  particular  contact  if  that  contact  moved 
in  the  previous  model  to  the  localization  server,  it  would  now  move  to 
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it  as  still  a  possible  sub  with  a  certain  probability,  or  as  a  certain 
sub  with  another  probability,  or  it  may  exit  into  the  false  contact  bin 
with  the  remaining  probability.  Similarly,  at  the  completion  of  a  locali¬ 
zation  service,  contacts  would  split  off  into  different  categories. 
Presumably,  the  probability  of  classification — i.e.,  passing  from  the 
possible  sub  to  either  the  certain  or  nonsub  states — would  be  modeled 
with  a  higher  and  higher  probability  as  a  contact  progresses  further  and 
further  into  the  queueing  system.  This  model  is  relatively  easy  to  imple¬ 
ment  by  simulation,  and  initial  analytical  effort  has  been  made  to  set  up 
the  appropriate  Markov  model. 

Model  2 — An  alternative  approach  is  to  assume  that  classification 
is  a  process  in  which  the  decisions  are  made  at  separate  times  from  the 
times  of  a  contact's  transition  from  one  queueing  service  to  another  in 
the  system.  One  way  to  handle  this  is  to  regard  classification  as  being 
performed  by  a  separate  server  whose  completion  of  service  competes  with 
others  in  the  Markov  model  for  determination  of  the  next  transition.  In 
practice,  classification  is  really  accomplished  jointly  by  the  personnel 
at  the  various  ASW  function  locations,  and  presumably  a  final  decision 
rests  on  the  commanding  officer  who  is  not  associated  expressly  with  any 
one  of  the  servers  of  our  Markov  model  and  receives  information  from  all 
of  them.  Thus,  it  is  logical  to  consider  an  additional  "competing  process" 
that  contributes  to  the  Markov  model's  transition  decisions  and  includes 
classification  (Appendix  A  of  Ref.  1). 

Because  the  second  model  is  basically  no  more  difficult  to  simulate 
than  the  first  one,  it  was  chosen  for  extending  the  currently  available 
simulation  model.  A  detailed  set  of  flow  charts  to  implement  this  ap¬ 
proach  were  prepared  but  not  programmed  because  of  the  change  of  emphasis 
toward  models  that  are  more  easily  related  to  operational  parameters. 
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Although  the  single  ASW  unit  model  with  classification  added  was 
not  carried  through  to  a  computational  analysis,  recording  its  conceptual 
attributes  is  worthwhile.  The  structure  of  either  Model  1  or  Model  2 
above  is  depicted  in  Figure  2.8.  The  "Black  Box"  modeling  the  classifica¬ 
tion  process  will  be  explained  further  below.  The  dotted  lines  represent 
the  direction  of  effect  of  any  classification  decision.  The  assumption 


FIGURE  2.8  STRUCTURE  OF  SINGLE-UNIT  MODEL  INCLUDING  CLASSIFICATION 


was  made  that  a  decision  to  classify  a  contact  nonsub  immediately  rejects 
the  contact  out  of  the  system  so  that  other  contacts  can  move  up  in  their 
queues.  A  decision  that  some  contact  is  certain  sub  has  been  tentatively 
assumed  to  have  the  possible  effects  shown  in  Table  2.2. 

The  features  shown  in  the  table  model  possible  effects  of  a  classi¬ 
fication  decision  on  the  operation  of  the  basic  single-unit  model.  How¬ 
ever,  considering  the  reverse  effect  of  system  state  transitions  on  the 
classification  process  itself  may  also  be  important.  First  of  all,  a 
model  of  the  structure  of  the  "Black  Box"  in  Figure  2.8  must  be  chosen. 
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Table  2.2 


EFFECTS  OF  "CERTAIN"  SUBMARINE  CONTACTS 


Case 

Result 

A  new  certain  sub  contact 

The  not-yet-classif ied  contact  is  displaced 

in  a  queue  is  held  up  by 

by  the  new  certain  sub  contact,  and  the 

a  not-yet-classif ied 

former  goes  back  into  queue,  with  all 

contact  in  next  service. 

service  to  date  invalidated — i.e.,  it  will 

start  from  scratch  the  next  time  it  enters 

that  service. 

New  certain  sub  contact 

The  service  rate  may  possibly  be  raised, 

is  in  a  service. 

thus  indicating  more  concerted  effort  by 
personnel. 

In  the  investigations  by  Operations  Research  Incorporated  (ORI)  in  a 

parallel  effort  related  to  this  project,3  the  classification  process  was 

basically  represented  by  a  "growth  of  information"  about  a  contact  versus 

statistical  sampling  of  that  information  to  arrive  at  a  classification 

decision.  A  simplified  view  of  this  approach  was  adopted  by  representing 

growth  of  information  as  shown  in  Figure  2.9.  The  growth  transition  rate 

A.  would  naturally  be  higher  than  the  regression  rate  p.  in  Figure  2.9, 
c  c 

and  would  thereby  ensure  that  a  contact  would  eventually  be  classified 
(if  it  stayed  in  the  system  long  enough).  The  behavior  of  the  model  in 
Figure  2.9  would  be  approximately  like  that  of  an  "exploding"  queue  with 
intensity  p  =  \/p  >  1,  in  which  the  occurrence  of  a  classification  would 


FIGURE  2.9  A  MARKOV  MODEL  OF  THE  CLASSIFICATION  PROCESS 
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be  represented  by  an  overflow  from  the  truncated  queue.  This  model  might 
even  be  adaptable  to  including  queueing  features  separate  from  those  of 
the  rest  of  the  model. 

Additional  statistical  measures  expected  to  be  derivable  from  the 
unit  ASW  model  with  classification  added  are: 

•  The  distribution  of  time  a  true  contact  spends  in  the  system 
for  a  given  false  contact  ratio  at  the  detector  input. 

•  The  relative  degrees  of  congestion  at  various  parts  of  the 
system  caused  by  presence  of  false  contacts. 
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Ill  CONDITIONAL  PROBABILITY  MODELS 


A.  Introduction 

Basically,  two  types  of  effectiveness  models  are  used  in  ASW  studies. 
The  first  type  of  model — often  called  an  analytical  model — is  used  to 
estimate  (or  predict)  effectiveness  when  basic  parameters  of  the  problem 
such  as  detection  ranges,  sweep  widths,  environmental  variables,  and  so 
on,  are  known.  Equations  (mathematical  or  mathematical  and  logical)  are 
given  from  which  system  effectiveness  may  be  calculated  when  these  input 
quantities  are  known  or  are  assumed  to  be  known.  The  ASW  operations 
modeled  may  never  have  actually  been  performed  (nonexistent  equipment 
may  be  included)  so  that  data  are  unavailable  for  estimating  many  of  the 
model  parameters.  The  usual  procedure  is  to  use  the  available  data  to 
estimate  the  input  variables  of  the  model — which  often  characterize 
equipment  or  systems — and  to  use  these  data  also  to  structure  the  model 
wherever  possible,  and  to  hypothesize  the  remainder  of  the  structure. 

The  effectiveness  measure  for  this  kind  of  model  is  often  a  probability 
of  success,  such  as  the  probability  of  a  destroyer  detecting  a  submarine, 
or  a  submarine  detecting,  classifying,  closing,  and  successfully  attacking 
another  submarine. 

The  second  type  of  model — often  called  an  operational  model — also 
requires  inputs,  but  these  are  directly  observed  quantities  from  real 
ASW  operations  or  from  ASW  exercises.  Instead  of  assuming  values  of  in¬ 
put  variables  and  computing  intermediate  quantities  such  as  conditional 
probabilities,  an  operational  model  estimates  them  directly  from  observed 
outcomes . 
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A  common  form  of  model  used  both  as  an  analytical  and  operational 
model  is  the  conditional  probability  model.  The  basic  idea  is  to  choose 
a  series  of  requirements  necessary  and  sufficient  for  mission  success. 
These  requirements  often  are  directly  related  to  equipment  functions  or 
system  or  subsystem  functions;  often,  an  interface  of  a  subsystem  with 
another  subsystem  is  where  achievement  of  a  requirement  is  to  be  observed. 
Because  the  achievement  of  a  requirement  is  necessarily  conditional  on  an 
opportunity  for  the  portion  of  the  system  to  meet  the  requirement  and  it 
is  often  assumed  to  be  a  probability,  this  form  of  model  is  known  as  a 
conditional  probability  model. 

B.  A  Conditional  Probability  Model  as  a  Markov  Model 

Conditional  probability  models  can  be  conveniently  described  by  a 
simple  diagram  made  up  of  branches  and  nodes.  These  branches  correspond 
to  requirements  and  the  nodes  correspond  to  cumulative  requirements  met. 

An  example  of  a  typical  conditional  probability  model  of  the  opera¬ 
tional  type  is  a  simplified  version  of  the  WSE  model  developed  and  used 
in  the  Big  Daddy  series  of  sub-on-sub  barrier  exercises  performed  several 
years  ago.  The  diagram  for  this  model  is  shown  in  Figure  3.1.  The  labels 
within  the  nodes,  called  states,  are  oversimplified  descriptions  of 
possible  stages  in  the  sequence  of  events  leading  to  kill.  More  complete 
definitions  are  as  follows. 


Opportunity 

A  detection  opportunity  is  presented. 

Detect 

To  detect  the  transmitting  submarine  given  a  detec¬ 
tion  opportunity. 

Classify 

Correctly  classify  a  submarine  as  a  submarine,  given 
that  a  submarine  has  been  detected. 

Attack 

Satisfy  attack  conditions  on  the  submarine,  given 
that  the  submarine  was  correctly  classified. 

Kill 

Submarine  is  destroyed,  or  placed  out  of  action, 
given  that  an  attack  was  made. 
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FIGURE  3.1  STATES  IN  THE  WEAPON  SYSTEM  EFFECTIVENESS  MODEL 


The  arc  (or  branch)  from  detect  to  classify  in  a  sense  represents 
the  requirements  for  classification  because  a  transition  from  detect  to 
classify  is  made  only  when  the  classification  requirements  are  met.  Other 
branches  may  be  similarly  interpreted  by  consideration  of  the  nodes  (or 
states)  they  connect. 

The  diagram  of  Figure  3.1  can  be  modified  by  adding  a  node  (state) 
labeled  fail,  for  failure  to  kill  the  transiting  submarine.  Since  failures 
can  occur  by  failing  to  detect  given  opportunity,  to  classify  given  de¬ 
tection,  and  so  on  in  sequence,  branches  can  be  added  to  show  transitions 
from  individual  states  to  Fail  as  shown  in  Figure  3.2. 


FIGURE  3.2  TRANSITION  DIAGRAM  FOR  THE  WEAPON  SYSTEM  EFFECTIVENESS  MODEL 


In  terms  of  the  diagram  in  Figure  3.2,  a  given  barrier  exercise  re¬ 
sulting  in  a  detection  opportunity  can  be  summarized  by  specifying  the 
path  followed  from  the  opportunity  state  to  one  of  the  terminal  states 
kill  or  fail.  If  states  are  represented  by  their  first  letters,  then  a 
string  of  symbols  representing  the  path  can  be  determined  from  each  exer¬ 
cise.  For  example,  the  path  ODCF  means  the  transiting  submarine  presented 
a  detection  opportunity,  it  was  then  detected  and  classified  by  the 
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barrier  submarine,  but  attack  was  not  accomplished;  hence,  the  transiting 
submarine  failed  in  its  mission.  The  desired  path  for  the  barrier  sub¬ 
marine  is  the  unique  path  from  opportunity  to  kill  represented  by  ODCAK. 

The  measure  of  effectiveness  for  the  barrier  submarine  mission  (WSE) 
is  now  defined  as  the  probability  of  kill  of  a  transiting  submarine, 
given  that  a  detection  opportunity  is  presented. 

By  the  use  of  standard  formulas  for  conditional  probability,  the 
probability  of  kill  given  opportunity  can  be  written: 


P(K|0)  =  P(D|0)  X  P(c|0  and  D)  X  P(a|o  and  D  and  C) 

X  P(K)0  and  D  and  C  and  A)  .  (3.1) 

By  assumption,  however,  detect  can  only  occur  following  opportunity,  and 
classify  can  only  occur  following  detect.  Therefore,  the  only  remaining 
conditions  in  each  conditional  probability  are  the  latest  ones,  so  that 
the  formula  for  WSE  becomes: 

WSE  =  P(K|0)  =  P(DjO)  X  P(C|D)  X  P(A|C)  X  P(K|A)  .  (3.2) 

In  terms  of  the  state  diagram  of  Figure  3.2,  WSE  is  the  probability 
that  a  path  terminates  in  kill,  given  that  it  started  in  opportunity. 

If  the  conditional  probabilities  of  formula  (3.2)  are  recorded  on  the 
branches  of  Figure  3.2  as  shown,  WSE  is  seen  to  be  the  product  of  the 
probabilities  along  the  path  from  opportunity  to  kill.  The  value  of  WSE 
is  estimated  from  exercise  data,  where  each  conditional  probability  is 
estimated  by  a  ratio — number  of  transitions  to  next  state:  number  of 
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opportunities  for  this  transition.  For  example,  the  estimate  of  P(c|D) 
is : 

number  of  classifications  ,  .  „ 

- * - T—- — “ -  =  P(C  D)  .  (3.3) 

number  of  detections  1 

Now  assume  that  data  from  a  number  of  transit  exercises  are  avail¬ 
able,  and  there  are: 

N  transits  presenting  detection  opportunity 
n^  transits  resulting  in  detection 
n^  transits  resulting  in  classification 
n^  transits  resulting  in  attack 
n  transits  resulting  in  kill. 

The  diagram  of  Figure  3.2  can  be  used  to  tabulate  these  numbers  of  transi¬ 
tions,  and  the  result  is  shown  in  Figure  3.3.  The  values  at  the  nodes  are 
the  numbers  of  paths  entering  the  given  node. 


FIGURE  3.3  DATA  FOR  THE  WEAPON  SYSTEM  EFFECTIVENESS  MODEL 


Replacing  each  conditional  probability  in  the  definition  of  WSE  by 
its  estimate  results  in  the  following  estimate  of  WSE. 
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Because  the  n  ,  n  ,  and  n  terms  all  cancel,  the  end  result  is 
1'  2’  3  ’ 

n4  number  of  kills 

WSE  =  —  =  -  .  (3.5) 

N  number  of  transits  presenting  opportunity 

If  a  transit  presenting  an  opportunity  is  considered  a  trial,  and  a  kill 
is  a  successful  trial,  then 

number  of  successes 

WSE  =  -  (3.6) 

number  of  trials 

which  is  the  usual  estimator  for  the  probability  of  success  for  Bernoulli 
trials.  If  the  assumption  can  be  accepted  that  a  set  of  transit  exer¬ 
cises  can  be  considered  a  set  of  independent  Bernoulli  trials,  then 
simply  counting  the  number  of  successes  would  provide  this  estimate  of 
WSE  directly. 

The  intermediate  states  (detect,  classify,  attack)  did  not  matter 
because  the  transition  counting  entailed  in  passing  through  these  states 
cancelled  out.  This  cancellation  will  always  occur  in  this  kind  of  model. 
Further  discussion  of  this  point  will  be  deferred  to  Section  VII.  At 
this  point,  it  suffices  to  note  that  the  cancellation  occurs  when  each 
transit  begins  with  opportunity  and  ends  with  kill  or  failure,  provided 
that  no  repeated  entries  occur  in  any  state. 

The  assumption  was  made  above  that  each  exercise  could  be  translated 
into  a  path  from  the  starting  node  to  either  of  the  terminal  nodes  without 
returning  to  any  state  after  leaving  it.  In  terms  of  the  diagram,  this 
means  that  no  path  formed  a  "loop."  If  one  attempts  to  calculate  success 
probabilities  in  a  problem  with  loops  by  the  usual  conditional  methods, 
the  results  will  be  in  error. 

An  example  of  an  erroneous  attempt  is  shown  in  Figure  3.4,  where 
the  detect  and  classify  states  from  Figure  3.2  have  been  merged  for 
simplicity. 
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PATH 

NUMBER 

OBSERVED  PATH 

NUMBER  OF 
TIMES  OBSERVED 

(1) 

0  0  A  K 

"l 

ODAF 

n2 

O  D  F 

n3 

(4) 

0  F 

n4 

(5) 

ODODAK 

n5 

FIGURE  3.4  SUCCESS  PROBABILITIES  WITH  LOOPS  (BASED  ON  FIGURE  3.3) 


Assume  that  the  observed  paths  and  the  numbers  of  times  each  path 
was  observed  are  recorded  as  shown  on  the  bottom  of  Figure  3.4.  For 
example,  n^  paths  are  designated  0  D  A  K,  meaning  that  there  were  n^ 
occurrences  of  the  sequence  Opportunity  -»  Detect  -*  Attack  ->  Kill.  A 
return  to  an  earlier  state  (a  loop)  occurs  in  the  last  path  since  oppor¬ 
tunity/detect  occurs  more  than  once  in  this  path.  Because  paths  1  and  5 
terminate  in  kill,  the  number  of  successes  is  (n^  +  n  )  and  the  Bernoulli 
estimator  of  probability  of  kill  P  (K|0)  is 


Pb(K|0) 


IX 

■i 


(3.7) 


In  the  path  of  type  5,  the  opportunity  state  is  entered  twice.  That 
is,  following  an  initial  opportunity  is  a  first  detection,  then  the  con¬ 
tact  is  lost,  and  a  second  opportunity  is  presented.  This  then  presents 
two  opportunities. 


45 


The  number  of  possible  transitions  between  all  pairs  of  states  can 
be  conveniently  recorded  on  the  branches  between  states  as  shown  in 
Figure  3.4.  To  obtain  these  values,  each  path  through  from  opportunity 
to  kill  or  fail  can  be  traced  and  added  to  the  counter  on  each  branch  as 
the  transition  is  made,  or  the  number  of  transitions  from  the  encoded 
path  definitions  can  simply  be  counted. 

Consider  next  the  usual  conditional  probability  estimator  for  proba¬ 
bility  of  Kill. 


p(k|o)  =  p(d|o)  x  p(a|d)  x  p(k|a) 

number  of  number  of  number  of 

detections  attacks  kills 

=  -  x  -  x - 

number  of  number  of  number  of 

opportunities  detections  attacks 


Using  transition  data  from  Figure  3.4  and  estimating  each  fraction  as 


(number  of  transitions  out  of  a  state  to  designated  state) 


results  in  (total  number  of  transitions  out  of  the  state) 


number  of  detections 
number  of  opportunities 

number  of  attacks 
number  of  detections 

number  of  kills 
number  of  attacks 


n  +  n  +  n  +  2n 
12  3 _ 5 _ 

n  +  n  +  n  +  n  +  2n 
1  2  3  4  5 

n  4-  n  +  n 
12  5 


n  +  n  +  n  +  2n 
12  3  5 

n  +  n 
1  5 


n  +  n  +  n 
12  5 


Putting  these  ratios  into  Eq.  (3.8)  shows  that  the  number  of  detec¬ 
tions  and  attacks  again  cancel,  resulting  in 


p(k|o) 


_ number  of  kills _ 

number  of  opportunities 

/ 


n 

1 


+  n 

5 


n  +  n  +  n  +  n  +  2  n 
1  2  3  4  5 


(3.9) 
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The  numerator  of  Eq.  (3.9)  agrees  with  that  in  Eq.  (3.7),  but  the 

denominator  of  Eq.  (3.9)  is  larger  than  that  in  Eq.  (3.7)  by  n  ,  which 

5 

is  the  number  of  second  opportunities.  Because  n  is  nonnegative,  the 

5 

Bernoulli  estimator  of  Eq.  (3.7)  cannot  be  less  than  the  usual  estimator 
of  Eq.  (3.8);  that  is, 


P  (K|0)  £  p(k|o) 


(3.10) 


Therefore,  the  usual  estimator  of  Eq.  (3.8)  is  incorrect;  it  esti¬ 
mates  a  different  quantity  from  P(K|o).  The  Bernoulli  estimator  is  the 
correct  estimator  to  use  in  this  situation,  and  its  formula  in  terms  of 
estimates  of  conditional  probabilities  will  be  derived  by  Markov  methods 
in  Appendix  C  on  Mason's  rule.  The  formula,  given  here  without  justifi¬ 
cation,  is 


P(K | 0) 


P(D|0)  P(A|D)  P(K|A) 

l  -  p(d|o)  p(o|d) 


(3.11) 


The  new  conditional  probability  in  the  denominator  P(o|d)  is  the  condi¬ 
tional  probability  associated  with  the  detect-to-oppor tuni ty  branch. 

That  is,  it  is  the  probability  of  another  opportunity  following  detection 
(as  contrasted  with  attack  or  fail),  conditioned  on  detection  occurring. 

Still  other  loopless  estimators  may  be  attempted  with  conditional 
probability  methods.  For  example,  the  return  loop  from  detect  to  oppor¬ 
tunity  may  be  simply  ignored.  The  ratios  estimating  P(D|0)  and  P(K|A) 
remain  the  same,  but  P(a|d)  changes  to 

n  +  n  +  n 

~  .  12  5 

P(A  D)  =  7 - v - 

'  /n+n+n\  +  n 

\  1  2  5}  3 

because  the  n  returns  to  opportunity  are  ignored. 

5 
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Substitution  of  this  value  in  Eq.  (3.8)  gives  a  new  estimator 


P(KjO) 


(n  +  n  +  n  +  2  n  |  (n  +  n\ 

_ \  1  2  3  5)\l  5} 

7n  +  n  -f  n  +  n  +  2  n  \["/n +  n  +  n  \  +  n  "1 

\  1  2  3  4  5)l[  1  2  5)  3j 


which  is  again  incorrect  because  it  differs  from  the  Bernoulli  estimator. 


C.  Limitations  of  Conditional  Probability  Models 


It  was  just  observed  that  the  probability  of  success  estimator 
arrived  at  by  successive  conditioning  to  form  a  conditional  probability 
model  is  identical  to  the  estimator  for  Bernoulli  trials  when  each  transit 
exercise  started  from  the  same  state  (opportunity)  and  ended  in  success 
(kill)  or  fail.  (Exercise  data  of  this  type  will  be  called  "complete" 
exercise  data  henceforth.)  Further  limitations  of  conditional  probability 
models  are: 

•  Only  a  limited  number  of  situations  are  defined — usually 
one-on-one  engagements.  Applicability  is  limited  to  tac¬ 
tical  situations  in  which  a  single  set  of  serial  require¬ 
ments  for  mission  success  can  be  identified. 

•  They  are  static ,  and  therefore  problems  in  which  time  is  an 
essential  element  may  not  be  capable  of  being  treated. 

Dynamic  operational  data  (such  as  average  time  to  prosecute 
a  false  contact)  cannot  be  directly  used  in  estimating  the 
parameters  of  a  conditional  probability  model. 

•  They  have  few  if  any  controllable  variables.  Hence,  they 
are  of  limited  use  in  improving  equipment  or  operations. 

They  cannot  be  used  to  improve  tactics. 

•  Degrading  factors  such  as  false  contacts  do  not  appear  ex¬ 
plicitly.  Hence,  the  quantitative  influence  of  these  de¬ 
grading  factors  cannot  be  assessed. 

•  Only  a  limited  portion  of  the  available  data  is  used  for 
parameter  estimation  because  only  static  transition  data 
can  be  used  to  estimate  their  parameters. 

•  They  have  limited  predictive  power. 

•  Confidence  limits  are  difficult  to  obtain. 
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In  view  of  these  limitations,  the  following  justify  the  use  of  a 
conditional  probability  model. 

•  Conditional  probability  models  incorporate  all  the  pertinent 
operational  and  environmental  factors  directly  and  therefore 
require  a  minimal  number  of  hypothesized  relationships. 

•  Data  are  often  available  from  sources  other  than  complete 
exercises.  In  particular,  data  are  often  available  from 
truncated  exercises  (those  beginning  in  an  intermediate 
state/ending  in  an  intermediate  state). 

•  The  value  of  the  probability  of  success  is  not  the  only 
useful  information.  For  example,  breaking  down  an  operation 
into  phases  and  observing  transition  information  as  is  done 
in  estimating  transition  probabilities  often  reveals  points 
where  system  operation  may  be  improved. 

•  Some  transitions  may  be  unobservable,  and  complete  exercises 
may  therefore  be  impossible.  For  example,  live  torpedoes 
cannot  be  fired  at  exercise  submarines  so  that  the  attack- 
to-kill  transitions  cannot  be  directly  observed.  Data  from 
other  sources,  such  as  simulation,  may  have  to  be  used  in 
estimating  some  transition  probabilities. 

•  The  individual  conditional  probability  estimates  may  be 
studied,  analyzed,  or  estimated  as  a  function  of  parameters 
of  the  mission/environment. 
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IV  MARKOV  AND  SEMI-MARKOV  MODELS  IN  ASW  EFFECTIVENESS  ANALYSIS 


A.  Introduction 

In  the  previous  section,  conditional  probability  models  were  dis¬ 
cussed  and  analyzed,  and  it  was  pointed  out  that  a  conditional  probability 
model  can  be  considered  a  special  case  of  a  Markov  model.  Moreover,  some 
of  the  limitations  on  conditional  probability  models  can  be  removed  by 
considering  more  general  Markov  models. 

In  this  section,  some  additional  general  reasons  for  considering 
Markov  (and  semi-Markov)  models  for  ASW  effectiveness  analysis  are  first 
given  in  outline  form.  Following  this  is  a  discussion  of  the  measures  of 
effectiveness  that  a  Markov  model  may  estimate.  Finally,  the  steps  en¬ 
tailed  in  formulating  and  using  a  Markov  model  and  in  incorporating  one 
or  more  of  these  measures  of  effectiveness  are  briefly  discussed.  Later 
sections  in  the  report  deal  more  fully  with  model  formulation  and  parameter 
estimation. 

B.  Useful  Properties  of  Markov  Models 

In  addition  to  removing  some  of  the  limitations  on  conditional  proba¬ 
bility  models,  the  reasons  for  considering  Markov  models  in  ASW  effective¬ 
ness  analysis  are: 

•  Markov  models  are  general  and  flexible  in  the  following  ways. 

-  Many,  perhaps  most,  measures  of  effectiveness  used  in  naval 
warfare  studies  and  analyses  can  be  formulated  in  terms  of 
an  appropriate  Markov  model.  (Conditional  probability 
models  are  a  special  case  of  a  Markov  model.)  New  measures 
of  effectiveness  can  also  be  formulated  that  may  be  useful 
where  effectiveness  has  remained  elusive. 
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-  Markov  models  can  be  used  at  various  levels,  ranging  from 
one-on-one  engagements/encounters  to  the  task  force  level. 

-  Markov  models  offer  a  way  to  combine  data  from  diverse 
sources  in  a  single  model.  Measures  of  effectiveness  at 
various  levels  can  all  be  introduced  into  a  single  model. 

•  Markov  models  are  fundamentally  dynamic  in  nature,  but  the 
dynamic  and  static  aspects  conveniently  factor  to  allow  the 
dynamic  aspects  to  be  analyzed  separately.  Shifting  from 
static  to  dynamic  modeling  presents  new  opportunities  to 
exploit  operational  data  that  are  currently  not  used  in 
effectiveness  modeling. 

•  For  certain  of  the  measures  of  effectiveness,  Markov  models 
show  a  remarkable  insensitivity  to  the  actual  nature  of 
the  underlying  random  process.  That  is,  they  are  very 
"robust"  in  the  statistical  sense. 

•  Markov  models  are  computationally  convenient. 

-  Analytical  (closed-form)  solutions  can  be  found  for  small 
models . 

-  Systematic  computational  procedures  exist  for  determining 
-all  of  the  reasonable  measures  of  effectiveness  commonly 
used  for  Markov  models  of  any  reasonable  size. 

-  Many  dynamic  properties  of  the  model  can  be  routinely 
calculated. 

•  Many  systematic  methods  exist  for  estimating  Markov  model 
parameters  from  observed  data.  Several  of  these  methods 
employ  data  not  ordinarily  used  in  ASW  effectiveness 
modeling. 

•  Markov  models  may  be  useful  for  comparing  dissimilar  models 
and  for  validating  simulation  models. 


C.  Measures  of  Effectiveness 

Four  principal  measures  of  ASW  mission  effectiveness  will  be  con¬ 
sidered  in  this  report.  All  of  these  measures  can  be  defined  in  terms 
of  a  Markov  process.  Many,  or  perhaps  most,  measures  of  effectiveness 
commonly  used  in  naval  warfare  analysis  studies  are  one  of  these  measures. 
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Probability  of  Mission  Success — In  this  first  measure  of  effective¬ 


ness,  success  must  be  suitably  defined  for  the  mission  being  considered. 
Classical  conditional  probability  models  used  in  ASW  analysis  are  of  this 
type,  where  success  often  means  kill  of  the  enemy  submarine.  In  other 
contexts,  success  may  be  simply  detection  of  the  enemy  submarine,  or  de¬ 
tection  and  classification  of  the  enemy  submarine.  This  measure  is  funda¬ 
mentally  a  static  measure. 

To  incorporate  this  measure  into  a  Markov  model,  a  state  labeled 
Success  must  be  introduced  and  made  an  absorbing  (trapping)  state.  The 
probability  of  success  will  then  be  the  probability  of  the  process  being 
absorbed  in  the  success  state.  This  probability  depends  on  the  static 
structure  of  the  Markov  process  rather  than  on  its  dynamics. 

Mean  Time  to  Success  (or  Mean  Time  to  Fail)  —  In  this  second  measure 
of  effectiveness,  for  example,  an  aircraft  carrier  in  the  objective  area 
may  be  considered  to  fail  in  its  mission  when  an  enemy  submarine  detects 
it.  The  mean  time  to  detection  of  the  carrier  is  an  appropriate  measure 
for  many  studies. 

In  Markov  process  terms,  the  success  (or  fail)  state  is  required  as 
above,  and  success/fail  are  again  made  absorbing  states.  Methods  for 
calculating  the  mean  time  to  absorption  in  a  designated  trapping  state 
are  available  from  Markov  theory.  The  dynamics  of  the  Markov  model  are 
entailed  in  the  calculation  of  this  measure. 

Probability  of  Success  (or  Fail)  at  or  by  a  Specified  Time — In  this 
third  measure  of  effectiveness,  if  an  aircraft  carrier  can  be  considered 
.  to  succeed  in  its  mission  after  no  detection  for  (say)  ten  days,  then  the 
chosen  measure  of  effectiveness  may  be 

P(carrier  is  not  detected  in  ten  days) 
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Again  the  Markov  formulation  requires  appropriately  defined  success/ 
fail  states  that  are  absorbing  states.  The  measure  becomes 

P(process  is  not  absorbed  in  the  fail  state  by  time  T) 

where  T  is  the  specified  duration.  This  measure  again  depends  on  the 
dynamics  of  the  Markov  model. 

Probability  of  Occupying  a  Designated  State  at  a  Randomly  Chosen 
Instant  of  Time — This  fourth  measure  of  effectiveness  is  appropriate  in 
situations  where  mission  success  cannot  be  identified  with  an  absorbing 
state,  as  in  a  trailing  mission. 

In  contrast  to  the  Markov  formulations  of  the  preceding  three 
measures,  this  measure  is  defined  in  terms  of  the  steady-state  (as  opposed 
to  transient)  properties  of  the  Markov  process.  The  measure  again  depends 
on  the  dynamics  of  the  process. 

The  first  of  these  measures  is  a  static  measure  and  the  other  three 
are  dynamic.  However,  the  static  measure  is  imbedded  within  the  dynamic 
Markov  model  so  that  all  are  computable  from  a  single  dynamic  model.  In 
many  contexts,  the  first  three  measures  may  all  be  incorporated  within  a 
single  Markov  model. 

When  a  Markov  model  is  applied  to  a  real  naval  operational  problem, 
the  assumptions  required  for  a  Markov  model  will  seldom,  if  ever,  be 
exactly  fulfilled.  The  four  measures  vary  markedly  in  their  sensitivity 
to  the  underlying  assumptions,  however.  The  probability  of  success 
measure  is  singularly  insensitive  to  the  actual  nature  of  the  process 
being  modeled.  The  second  measure,  the  mean  time  to  success  (or  fail), 
is  also  very  "robust."  The  reasons  for  this  insensitivity  (robustness) 
are  discussed  in  detail  in  Section  VII.  The  last  two  measures,  the 
probability  of  not  failing  by  time  T  and  the  probability  of  occupying  a 


54 


designated  state  at  a  randomly  chosen  point  in  time,  depend  much  more 
heavily  on  the  assumptions  required  for  a  Markov  process.  Both  measures, 
it  should  be  noted,  are  special  cases  of  the  probability  of  occupying  a 
state  at  an  arbitrary  time  t.  For  this  reason,  most  of  the  experimental 
work  discussed  later  in  the  report  has  been  devoted  to  finding  methods 
for  restructuring  the  state  space  and  estimating  the  model  parameters  to 
obtain  satisfactory  estimates  of  P(state)  as  a  function  of  time  t. 

D.  Formulating  a  Markov  Model 

1 .  Defining  a  Set  of  States 

A  Markov  model  basically  consists  of  a  set  of  states  and  rules 
for  determining  how  transitions  will  occur  between  states.  In  applica¬ 
tions  to  ASW  effectiveness  analysis,  the  most  basic  and  probably  the  most 
difficult  step  will  be  defining  the  set  of  states.  How  the  states  are 
defined  will  generally  depend  on  which  of  the  four  measures  of  effective¬ 
ness  is  being  considered.  A  goal  would  be  to  find  a  single  set  of  states 
that  is  adequate  for  all  four  measures. 

In  Markov  theory,  a  state  essentially  describes  a  possible  con¬ 
dition  of  the  process  being  modeled.  All  possible  conditions  must  be 
anticipated  and  states  defined  accordingly;  at  any  instant  of  time,  the 
Markov  process  must  be  in  some  state  but  not  be  in  more  than  one  state. 

In  operational  terms,  the  states  must  be  defined  so  that  the  state  can  be 
determined  for  each  point  in  time  when  what  actually  transpired  during 
the  ASW  operation  (or  exercise)  is  known.  Thus,  from  the  definition  of 
the  set  of  states  and  suitable  information  about  the  exercise  (say  from 
the  analysis  report),  a  sequence  of  states  can  be  determined  that  repre¬ 
sents  that  exercise.  This  sequence,  together  with  the  times  at  which 
transitions  between  states  were  made,  will  often  comprise  the  basic  data 
for  estimating  the  parameters  of  a  Markov  model. 
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A  special  feature  of  many  of  the  Markov  models  considered  in 
this  report  is  that  the  process  starts  in  a  single  state  that  can  be 
designated  the  starting  state.  For  the  first  three  measures  of  effec¬ 
tiveness,  there  must  also  be  a  set  of  terminal  (absorbing)  states;  often, 
the  states  of  success  and  fail  will  suffice.  In  ASW  operations,  a  natural 
starting  state  is  often  either  search  or  detection  opportunity. 

2.  Example  of  a  Set  of  States 

The  conditional  probability  model  (WSE  model)  discussed  in  the 
previous  section  had  states  denoted  opportunity,  detect,  classify, 
attack,  kill,  and  fail.  The  process  makes  a  transition  from  opportunity 
to  detect  when  detection  occurs,  and  the  state  remains  detect  until 
classification  (i.e,,  the  classify  state  is  entered)  or  failure  occurs. 

The  terminology  can  sometimes  be  misleading:  note  that  to  be  in  the  de¬ 
tect  state  means  that  detection  has  occurred  but  neither  classification 
nor  failure  has  yet  occurred. 

The  critical  part  of  the  definition  that  is  impossible  to 
satisfy  in  practice  is  the  requirement  that  knowledge  of  the  state  alone 
determines  how  the  transition  to  the  next  state  shall  be  made.  More 
specifically,  the  requirement  is  that  all  transitions  shall  be  made  at 
random  by  consulting  a  set  of  transition  probabilities  associated  with 
the  state  being  occupied.  Emphatically,  because  transitions  out  of  a 
designated  state  will  sometimes  be  made  to  one  state  and  sometimes  to 
another  in  no  way  implies  that  the  transitions  can  be  adequately  described 
by  a  set  of  probabilities  associated  with  the  occupied  state.  That  is, 
not  all  transition  behavior,  even  apparently  random  behavior,  can  be 
described  by  a  Markov  model. 

Later  sections  of  this  report  deal  with  the  problem  of  formu¬ 
lating  a  Markov  model  for  a  stochastic  process  whose  states  do  not 
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satisfy  the  Markov  property.  Various  attempts  are  made  to  restructure 
the  state  space  by  adding  auxiliary  states  so  that  the  new  set  of  states 
more  nearly  satisfies  the  Markov  requirements. 

3.  Estimation  of  Markov  Model  Parameters 

When  a  satisfactory  set  of  states  has  been  found,  with  or  with¬ 
out  the  aid  of  chi-squared  "goodness  of  fit"  tests,  the  next  step  is  to 
estimate  the  model  parameters  from  the  available  data.  Since  all  the 
measures  of  effectiveness  can  be  calculated  in  terms  of  the  model  param¬ 
eters,  the  model  construction  is  then  complete.  Parameter  estimation  is 
covered  in  more  detail  in  Section  VI,  and  the  calculation  of  the  measures 
of  effectiveness  from  estimated  parameters  are  detailed  in  Section  V. 

A  major  part  of  this  report  concerns  the  methods  for  estimating 
the  parameters  in  a  Markov  model  defined  to  determine  one  or  more  of  the 
measures  of  effectiveness  listed  above.  The  methods  must  assume  observable 
data  from  fleet  operations  and  exercises  as  their  basic  inputs.  The  ob¬ 
servable  data  need  not  be  limited  to  that  conventionally  used  in  static 
models  such  as  conditional  probability  models,  however.  Indeed,  one  of 
the  principal  benefits  of  a  change  from  static  to  dynamic  modeling  may 
be  that  potentially  more  data  are  available  from  operations  and  exercises 
with  which  to  construct  and  validate  effectiveness  models. 

The  parameters  of  a  conditional  probability  model  are  usually 
estimated  from  transition  data;  that  is,  from  counts  of  the  observed 
numbers  of  transitions  between  the  states  of  the  model.  Additional 
potential  data  available  for  estimating  the  parameters  of  a  dynamic 
Markov  model  include  such  quantities  as  the: 

♦  Average  time  each  state  is  occupied. 

•  Average  time  of  the  first  entry  into  each  state. 
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•  Average  length  of  time  spent  in  each  state  i  before 
its  transit  to  another  state  j. 

•  Mean  time  to  absorption  in  designated  absorbing  states. 

•  Probability  of  occupying  each  state  as  a  function  of 
time. 

In  more  sophisticated  applications,  the  variances  of  these  various  times 
may  be  used  as  well  as  the  mean  values.  The  equations  relating  many  of 
these  quantities  to  basic  parameters  of  the  process  are  linear  and  thus 
imply  that  suitably  modified  least  squares  methods  can  be  used  to  esti¬ 
mate  the  basic  parameters  when  the  equations  are  inverted.  Since  three 
of  the  four  principal  measures  entail  the  estimation  of  the  probability 
of  occupying  a  state  as  a  function  of  time,  special  emphasis  has  been 
placed  on  methods  for  estimating  parameters  to  make  P( state)  as  a  func¬ 
tion  of  time  adequately  fit  the  observed  data. 

The  complementary  side  of  the  shift  from  static  to  dynamic 
modeling  is  also  important.  A  dynamic  model  with  its  parameters  can  be 
used  to  predict  properties  of  the  system  or  operation  being  modeled. 
These  predictions  can  then  be  compared  with  observed  data  to  determine 
the  adequacy  of  the  model  and  to  suggest  changes  in  model  structure  or 
parameters.  Static  models  are  more  limited  in  this  respect;  with  less 
structure  in  the  model,  fewer  predictions  can  be  made. 
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V  BASIC  MARKOV  CHAIN  THEORY 


In  this  section,  the  basic  concepts  of  Markov  chains  are  summarized 
and  formulas  are  derived  for  computing  several  measures  of  effectiveness 
when  the  basic  parameters  (the  transition  probabilities)  of  the  Markov 
chain  are  known.  The  development  here  is  algebraic  and  uses  matrix  nota¬ 
tion  and  operations  for  brevity  and  compactness.  Other  derivations, 
often  carried  out  using  flowgraph  methods,  are  given  as  needed  in  later 
sections.  A  single  example  of  a  Markov  chain  with  two  transient  states 
and  two  absorbing  states  follows  the  theoretical  development. 

A.  States,  Transitions,  Paths,  and  Loops 

The  principal  elements  of  a  Markov  chain  (the  simplest  Markov 
process)  are  a  set  of  states  and  a  set  of  transitions  between  states. 

A  state  is  basically  the  description  of  a  possible  condition  of  a  system 
and  the  selection  of  a  set  of  states  is  a  crucial  and  often  difficult 
task  in  applications.  In  this  section,  the  states  are  simply  designated 
by  integers  1,  2,  ...,  N. 

States  must  be  defined  so  that  the  system  being  modeled  is  in  one 
and  only  one  of  the  states  at  any  given  time.  For  a  Markov  chain,  the 
transitions  between  states  (changes  of  state)  are  defined  at  integer 
times  n  =  1,  2,  3,  ....  By  convention,  the  process  starts  in  some  state 
(usually  generically  designated  as  state  i)  at  time  n  =  0  and  makes  its 
first  transition  to  a  state  j  at  time  n  =  1. 

The  Markov  chains  of  interest  here  have  one  or  more  absorbing  states 
which  are  states  that  are  never  left  once  they  are  entered.  Entry  into 
an  absorbing  state  is  associated  in  applications  with  the  end  of  the 
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process  since  nothing  of  interest  happens  afterward.  Absorbing  states 
in  this  report  are  generally  defined  by  mission  success  (the  success 
state)/mission  failure  (the  fail  state).  Other  states  are  transient 
states  by  definition.  Eventually,  if  the  process  continues  long  enough, 
some  absorbing  state  will  be  entered  to  end  the  process. 

A  transition  diagram  may  be  constructed  for  any  Markov  chain.  It 
consists  of  nodes  representing  the  states  and  branches  (or  arcs)  that 
represent  possible  transitions.  Figure  5.1  shows  an  example  with  four 
states — two  are  transient  and  two  are  absorbing.  For  concreteness, 
states  1  and  2 — the  transient  states — may  be  identified  with  a  surface 


FIGURE  5.1  TRANSITION  DIAGRAM  FOR  FOUR-STATE  MARKOV  CHAIN 


ship  searching  for  a  submarine  and  prosecuting  a  false  contact.  The 
success  state  (state  4)  may  represent  detection  of  a  submarine;  corre¬ 
spondingly,  the  failure  state  (state  3)  may  be  associated  with  failure 
to  detect.  As  shown  by  the  diagram,  transitions  take  place  between 
states  1  and  2  as  the  ship  first  searches,  then  prosecutes  a  false  con¬ 
tact,  then  returns  to  search  and  so  on.  Transitions  from  a  state  to 
itself  (self-transitions)  represent  waiting  times  or  holding  times  in  a 
state;  the  branches  on  the  diagram  showing  self-transitions  are  called 
self-loops.  A  path  from  one  state  to  another  consists  of  a  sequence  of 
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branches  from  one  state  to  another  and  may  be  of  various  lengths.  A 
path  from  a  state  back  to  itself  without  visiting  any  state  more  than 
once  is  called  a  loop.  Thus  the  path  from  state  1  to  state  2  and  back 
to  state  1  is  a  loop. 

B.  Transition  Probabilities 

Transitions  are  made  from  state  to  state  in  accordance  with  a  set  of 
transition  probabilities  that  are  the  only  numerical  parameters  of  a 
Markov  chain.  One  probability  is  defined  for  each  branch  of  the  transi¬ 
tion  diagram  because  the  lack  of  a  branch  implicitly  means  that  the  corre¬ 
sponding  transition  probability  is  zero.  These  probabilities  may  be 
arranged  in  matrix  form,  with  the  rows  being  associated  with  the  transi¬ 
tions  out  of  a  state  and  the  columns  being  associated  with  transitions 
into  a  state.  In  a  stationary  Markov  chain,  the  transition  probabilities 
do  not  depend  on  time  and  are  denoted  by  p  .  The  formal  definition  is: 

ij 

p  =  P  (state  at  time  n  is  j I  state  at  time  n  -  1  is  i)  for 
i  j  1 

all  states  i,  j  and  all  times  n  =  0,  1,  2,  .... 


C.  Partitioned  Form  of  the  Transition  Matrix 

The  transition  matrix  for  the  Markov  chain  in  Figure  5.1  is  denoted 
by  P,  with  elements  p.  ..  It  can  be  partitioned  according  to  transient 


ij 

and  absorbing  states  and  the  resulting 
the  identity  matrix  I.  That  is, 
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where  the  submatrices  Q  and  R  are  defined  by 

P13  P14 

(5.2) 

P  P 

23  *24_ 

and  I  is  a  2  X  2  identity  matrix. 

In  general,  Q  is  a  square  matrix  of  dimension  r,  where  r  is  the 
number  of  transient  states.  Matrix  R  has  r  rows  and  a  columns  (i.e., 
it  is  dimensioned  r  X  a),  where  a  is  the  number  of  absorbing  states. 

The  identity  is  dimensioned  a  X  a.  It  should  be  noted  that  rows  of  P 
are  associated  with  the  state  occupied  before  a  transition,  and  columns 
of  P  with  the  state  to  which  the  transition  is  made.  Thus,  the  submatrix 
Q  is  associated  with  transitions  from  transient  states  to  transient 
states,  and  the  submatrix  R  with  transitions  from  transient  states  to 
absorbing  states.  The  identity  matrix  I  is  associated  with  the  transi¬ 
tions  from  absorbing  states  to  absorbing  states.  Since  absorbing  states 
are  defined  as  states  that  are  never  left  once  they  are  entered,  the 
probabilities  are  unity  on  the  diagonal  and  zero  off  the  diagonal. 

D.  Multistep  Transition  Matrices  and  Absorption  Probabilities 

The  transition  matrix  P  defines  transition  probabilities  for  single 
transitions.  If  P  is  multiplied  by  itself,  the  transition  matrix  for 
two  consecutive  transitions  is  obtained  with  elements  denoted  by  a  super¬ 
script  "2"  to  denote  the  number  of  consecutive  transitions. 

p2  -  (pi2>)  •  (5-3> 

If  the  matrix  multiplication  is  carried  out  in  using  the  partitioned 
form  on  the  right  side  of  Eq.  (5.1),  the  result  is  expressible  in  terms 
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of  the  submatrices  of  P  which  have  the  same  dimensions  as  the  original 
partitioning.  Equation  (5.4)  gives  the  result: 


(I  +  Q)R 
I 


(5.4) 


In  the  example, both  identity  matrices  I  are  of  dimension  2;  in  general 
the  dimensions  differ. 

Successive  multiplication  of  P  by  itself  yields  transition  matrices 

for  larger  numbers  of  consecutive  transitions.  The  n-step  transition 
n 

matrix  P  is  readily  shown  to  have  the  partitioned  structure  shown  in 
Eq.  (5.5). 


(5.5) 


n 

This  form  reveals  the  process  dynamics.  The  submatrix  Q  represents 

those  series  of  n  transitions  that  remain  in  transient  states  for  all  n 

transitions.  In  cases  of  interest  in  this  report,  the  process  will  al- 

n 

ways  be  absorbed  in  an  absorbing  state  so  that  all  elements  of  Q  will 
approach  zero  as  the  number  of  transitions  n  becomes  infinite. 

The  other  submatrix  associated  with  starting  the  process  in  transient 
2  n-1 

states  is(I+Q+Q  +...+Q  )R.  Multiplying  in  the  R  and  expanding 

into  its  component  terms  gives  Eq.  (5.6). 

2  3  n-1 

R  +  QR  +  QR  +  QR  +  ...  4-Q  R  .  (5.6) 


The  first  term,  R,  represents  those  cases  where  absorption  occurs  on  the 
first  transition.  The  second  term,  QR,  represents  transitions  that 
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remained  in  the  transient  states  for  one  transition  and  were  absorbed  on 


the  second  transition.  Later  terms  represent  absorption  after  some  number 

of  transitions  from  the  starting  transient  state  to  other  transient 

k 

states,  followed  by  absorptions.  In  general,  Q  R  corresponds  to  absorp- 
.  s  st 

tion  on  the  (k  +  1)  transition — the  first  k  transitions  were  between 

.  st 

transient  states  and  the  (k  +  1)  transition  to  an  absorbing  state. 

By  means  entirely  analogous  to  those  used  to  sum  a  geometric  series, 
the  limit  of  the  sum  in  Eq.  (5.6)  can  be  shown  to  be  (I  -  Q)  ^R.  [The 
necessary  auxiliary  formula  is  shown  in  Eq.  (5.7): 


I+Q  +  Q2+...+Qn+...=  Qn  =  (I  -  Q)  1  .]  (5.7) 

n=0 

Therefore,  the  "infinite  step"  transition  matrix,  which  is  the  limit  of 
Eq.  (5.5)  as  n  becomes  infinite,  has  the  form  shown  in  Eq.  (5.8). 


n  oo 

lim  P  =  P 

n-*oo 


0 


(I  -  Q)  1R 
I 


(5.8) 


Only  the  last  a  columns  associated  with  absorbing  states  have  nonzero 

entries.  In  particular,  the  submatrix  ( I  -  Q)  ^R  gives  the  absorption 

,  th 

probabilities  for  all  starting  states.  That  is,  the  (i,k)  element  of 

-1  th 

(I  -  Q)  R  is  the  probability  of  being  absorbed  in  the  k  absorbing 

state,  given  that  the  process  started  in  state  i  at  time  zero. 


This  result  is  usually  obtained  by  setting  up  a  system  of  simul¬ 
taneous  linear  equations  in  the  unknown  absorption  probabilities.4  It 
will  be  convenient  to  denote  the  matrix  (I  -  Q)  R  by  A,  with  elements 
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E.  z  Transforms 

Many  quantities  besides  the  absorption  probabilities  a  just  found 

ik 

by  matrix  multiplication  can  be  obtained  by  using  z  transforms  that  are 
also  called  geometric  transforms.  The  z  transform  is  analogous  to  the 
Laplace  transform  but  is  easier  to  grasp  intuitively  because  the  variable 
z  is  associated  with  the  number  of  transitions  made  between  states. 

Some  additional  notation  is  needed  at  this  point.  Let  the  matrix 
f(n)  be  the  matrix  whose  elements  ©  (n)  are  the  probabilities  of  oc- 

ij 

cupying  state  j  after  n  transitions  (i.e.,  at  time  n)  when  the  process 
started  in  state  i  at  time  n  =  0.  Let  S(n)  be  the  state  at  time  n. 
Equation  (5.9)  defines  $(n),  and  Eq.  (5.10)  relates  $(n)  to  the  transi¬ 
tion  matrix  P. 
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Using  the  general  relationship  in  Eq.  (5.7)  after  replacing  Q  by  Pz  re¬ 
sults  in  Eq.  (5.13). 

$(z)  =  (I  -  Pz)_1  .  (5.13) 


Each  element  <J>  (z)  of  the  matrix  in  Eq.  (5.13)  is  a  function  of  z 

ij 

called  the  transmission  from  state  i  to  state  j.  From  Eq.  (5.7),  the 
th 

(i,j)  element  has  the  form  shown  in  Eq.  (5.14)  when  states  i  and  j  are 
distinct . 

,  .  ,  (2)  2  (3)  3  , 

<t>  (z)  =  p  z  +  p  z+p  z  +  ...  .  (5.14) 

ij  ij  ij  ij 

F.  Inversion  of  the  Matrix  (I  -  Pz)  and  Applications 

By  the  use  of  the  partitioning  shown  in  Eq.  (5.1),  the  matrix  I  -  Pz 
can  be  written  in  the  form  of  Eq.  (5.15). 


(5.15) 


The  inverse  of  I  -  Pz  is  easily  verified  by  direct  matrix  multiplication 
to  be  the  matrix  shown  in  Eq.  (5.16). 


(5.16) 


Each  of  the  matrices  in  the  top  row  is  useful  for  deriving  formulas  for 
calculating  effectiveness  measures.  It  should  be  noted  that  the 
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coefficient  of  z*1  in  the  expansion  of  ( I  -  Qz)  ^  is  the  matrix  Qn  that 
occupies  this  same  position  in  Eq.  (5.5). 

n 

Similarly,  the  coefficient  of  z  in  the  expansion  of 

(I  -  Qz) ~1Rz 
I  -  z 

,  2  n-1 

found  in  the  upper  right  position  is  ( I  +  Q  +  Q  +...+Q  )R,  which 

occupied  this  same  position  in  Eq.  (5.5).  The  (1  -  z)  term  appearing 
in  the  denominator  of  Eq.  (5.16)  has  the  function  of  accumulating  the 
probabilities  in  the  absorption  states.  When  this  factor  is  ignored  and 
z  is  set  equal  to  unity,  the  upper  right  hand  matrix  reduces  to 
( I  -  Q)  =  A,  the  matrix  of  absorption  probabilities  found  earlier  in 
Eq.  (5.8). 

,  -1 

The  elements  of  the  submatrix  (I  -  Q)  that  keeps  recurring  are 

,  „  th 

also  meaningful  quantities:  the  (i,j)  element  is  the  mean  number  of 
entries  into  a  transient  state  j  before  absorption  occurs,  given  that  the 
process  is  started  in  state  i. 

To  show  this,  let  (I  -  Q)  be  denoted  by  T,  with  elements  t.  .. 

Define  random  variables  x  (n)  and  e  by 

ij  ij 


x.  .(1)  = 


1  if  S(n)  =  j 


0  if  S(n)  f-  j 


(5.17) 


Ex.  .(n) 
ij 


(5.18) 


where  the  subscript  i  again  indicates  that  the  process  starts  in  state 


i  at  time  n  =  0.  From  Eq.  (5.18),  the  mean  value  of  e.  .  is 

ij 
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(5.19) 


n=0 


x.  .(n) 
ij 


Because  x  (n)  has  only  two  values  (unity  and  zero),  its  mean  is 

ij 


x. .(n) 
ij 


=  1  •  P[S(n)  =  j  S(0)  =  1]  +  0  •  P[S(n)  f  j  S(0)  =  1] 


„  ,  ,  (n) 

.  .(n)  =  p  .  . 
ij  ij 


(5.20) 


Therefore,  the  mean  number  of  entries  into  a  transient  state  j  before 


absorption  (e  )  is 
ij 


+  =£•> 


n=0 


U)  =  t 

ij  ij 


(5.21) 


by  Eq.  (5.14)  with  z  =  1  and  the  definition  of  t 

ij 
"t  h 

If  the  process  is  truncated  after  the  M  step  (instead  of  being 
allowed  to  continue  until  absorption  occurs),  the  mean  number  of  entries 
into  j  before  absorption  or  truncation  is 


M  M  00  CO 

—  —  V'  <n)  V'  (n)  (n) 

- .  .  (M)  =  >  x.  .(n)  =  >  p.  .  =  >  p.  -  >  p. 

Ij  Z— 4  IJ  Z— i  IJ  Z— 4  IJ  /L-J  ij 


n=0 


n=0 


n=0 


n=M+l 


.  (5.22) 


th 


Thus,  e  (M)  is  seen  to  be  the  i,j  element  of  the  matrix 
ij 


(i  -  QM+1j(I  -  Q)  1  =  (I  -  Q)"1  -  QM+1(I  -  Q)  1 


-1  /  M+l  M+2 

=  (I  -  Q)  -IQ  +  Q  + 


...) 


.  (5.23) 


Truncation  at  M  is  therefore  accomplished  by  premultiplying  the  funda- 

,  -1  „  „  M+l 

mental  matrix  (I  -  Q)  by  a  correction  factor  I  -  Q 
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In  the  untruncated  case,  the  average  number  of  transitions  before 
absorption  when  the  process  starts  in  state  i  can  be  denoted  by  e . .  Its 
value  is  obtained  by  summing  over  all  r  transient  states 

r 

e^  =  t.  ,  for  i  =  1,  2,  r  .  (5.24) 

j=l  ^ 

In  applications  later,  the  mean  number  of  entries  into  state  j  will 
be  associated  with  the  mean  time  spent  in  state  j  before  absorption, 
i.e.,  before  the  end  of  the  exercise.  One  parameter  estimation  method 

A 

will  be  developed  by  using  a  matrix  T  whose  elements  are  the  average 
total  time  spent  in  each  of  the  transient  states  during  the  exercise. 

For  calculation  of  a  useful  effectiveness  measure,  however,  a  different 
mean  time  is  needed:  the  mean  time  until  absorption,  given  that  absorp¬ 
tion  occurs  in  state  k.  This  quantity  e(i,k)  is  also  calculated  from 
the  elements  in  the  partitioned  form  of  (I  -  Pz)  1  but  is  somewhat  more 
difficult  to  derive. 


G.  Probability  Generating  Functions 

Returning  to  the  z  transform  of  the  one-dimensional  sequence  f(0), 
f(l),  f(2),  ...,  we  can  interpret  f(n)  as  the  probability  that  an  integer¬ 
valued  random  variable  X  assumes  the  value  n.  This  transform,  known  as 
the  probability  generating  function  of  the  random  variable  X,  is  defined 

by 


CO  CO 

f(z)  =  /  f(n)zn  =  /  P(X  =  n)zU 

u  u 


and  can  then  be  differentiated  with  respect  to  z  to  obtain 


CO  CO 

f'(z)  =  nf(n)zn  1  nP(X  =  n)z 

n=0  n=0 


(5.25) 


(5.26) 
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When  z  is  set  equal  to  unity,  the  result  is  the  expected  (mean)  value 
of  X: 


E(X) 


f'(z  =  1) 


y  ^  nP(  X  =  n) 
n=0 


(5.27) 


It  will  be  necessary  to  know  later  that  the  second  moment  of  X  and 

2 

hence  its  variance  (var  X,  or  cr  )  can  also  be  found  by  differentiating 

X 

f(z).  Specifically,  the  second  derivative  evaluated  at  z  =  1  equals 
2 

E(X  )  -  E(X) 

n) 

(5.28) 

Since  the  variance  of  X  is  defined  as  the  mean  of  the  square  of  X  minus 
the  square  of  the  mean  of  X,  we  have 


CO  CO  CO 

f "  ( z  =  1)  =  >  n(n  -  1)P(X  =  n)  =  n2P(X  =  n)  -  nP(X  = 

n=0  n=0  n=0 


£ 

n=0 

=  E^X2)  -  E(  X) 


var 


X  =  e(x2)  -  e2<x>  =  [e(x2)  -  E(X)J  +  [E(X)  -  E2(X)] 


=  f"(z  =  1)  +  f'(z  =  1)  -  [f'(z  =  1)]' 


(5.29) 


In  the  matrix  in  Eq.  (5.16),  the  individual  elements  of  the  sub¬ 
matrix  (I  -  Qz)  ^Rz/(1  -  z)  are  the  generating  functions  of  the  cumula¬ 
tive  distribution  of  the  number  of  transitions  before  absorption.  Two 
changes  must  be  made  to  use  these  elements: 

•  The  (1  -  z)  must  be  dropped  because  it  accomplishes  the 
accumulation;  dropping  the  (1  -  z)  leaves  the  conditional 
probability  generating  function  of  the  density  function  of 
the  number  of  transitions. 

•  The  appropriate  absorption  probability  aik  must  be  used  to 
divide  what  remains  after  the  factor  (1  -  z)  is  dropped  so 
that  the  conditioning  on  absorption  in  state  k  is  reflected. 
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th  -1 

Algebraically,  the  (i,k)  element  of  (I  -  Qz)  Rz  divided  by  a,  — 

a  k 

th  1 

the  (i,k)  element  of  (I  -  Q)  R — is  the  probability  generating  function 
of  the  number  of  transitions  before  absorption  in  k,  given  absorption  in 
k  and  starting  state  i.  The  first  derivative  of  this  generating  function, 
evaluated  at  z  =  1,  is  the  desired  measure  of  effectiveness  e(i,k). 

Differentiation  of  the  probability  density  function  may  be  carried 
out  in  matrix  terms  by  using  the  auxiliary  formula 


“•  (I  -  Qz)"1  =  (I  -  Qz)  XQ( I  -  Qz)  1 
dz 


(5.30) 


Differentiating  M(z)  =  (I  -  Qz)  Rz  as  a  product  results  in 


—  [M( z) )  =  (I  -  Qz)_1Q( I  -  Qz)_1Rz  +  (I  -  Qz)  1R 
dz 


(5.31) 


When  evaluated  at  z  =  1,  this  expression  reduces  in  a  few  simple  steps  to 


C(I  -  Qz)Rz]|  =  (I  -  Q)_1[(I  -  Q)_1r]  = 


(5.31) 


The  matrix  on  the  right  is  therefore  equal  to  the  product  of  the  funda¬ 
mental  matrix  T  =  (I  -  Q)  1  and  the  absorption  probability  matrix 
A  =  (I  -  Q)  ^R,  both  of  which  are  elements  in  the  top  row  of  the  parti¬ 
tioned  form  of  (I  -  Pz).  in  Eq.  (5.16). 

The  (i,k)th  element  of  Eq.  (5.32),  denoted  by  e(i,k),  is  the  mean 
time  to  absorption  state  k,  given  absorption  in  state  k  and  starting 
state  i  at  n  =  0 


e  (  i,  k)  = 


( i,  k)  ^element  of  (I  -  Q)  ~*~_(I  -  Q)  ^R_ 
P(absorption  in  state  k| starting  state  i) 


Eh 


a  /a 
j  Jk  ik 


(5.32) 
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The  equation  A  =  TR  =  (I  -  Q)  R  can  be  made  quite  intuitive  when 
t  is  known  to  represent  the  average  number  of  transitions  into  state  j 

ij 

before  absorption,  given  starting  state  i.  The  product  t  p  is  the 

ij  Jk 

mean  number  of  times  that  state  k  is  entered  from  state  j,  given  starting 
state  i.  The  sum  over  all  transient  states  j  corresponds  to  the  inner 
product  of  row  i  of  T  =  (I  -  Q)  and  the  column  of  R  that  represents 
state  k.  The  value  of  the  sum  is  the  mean  number  of  entries  into  state 
k,  given  starting  state  i.  Because  a  single  trial  is  implicit  in  the 
calculation  of  this  mean,  the  absorption  probability  itself  equals  the 
mean  number  of  entries: 

E(number  of  entries  into  kj starting  state  i)  = 

1  X  P(absorption  in  state  k| start  in  state  i) 

+  0  X  P(absorption  in  state  other  than  k| start  in  state  i) 

c-  * 

This  result  is  found  by  a  different  method. 

Another  measure  useful  in  some  contexts  is  the  probability  of  not 

.  .  .  th 

failing  before  a  specified  time.  The  (i,k)  element  of 

,  M-l. 

(I  +  Q  +  ...  +  Q  )R  is  the  probability  of  occupying  absorption  state 
k  at  time  M  (i.e,,  the  probability  of  being  absorbed  in  k  at  or  before 
time  M) .  Therefore,  the  probability  of  not  failing  by  time  M  is 

p  (M)  =  1  -  P(absorption  into  the  fail  state  f  by  M) 

th  /  M-l\ 

=  1  -  (i,f)  element  of  II  +  Q  +  ...  +  Q  J R  .  (5.33) 

* 

Reference  5,  Vol.  I,  Equation  4.1.117. 


Operationally  this  quantity  may  have  an  interpretation  such  as  the 
probability  of  surviving  (or  remaining  undetected)  for  at  least  M  time 
intervals . 

2  M-l 

The  sum  I+Q+Q  +  ...  +  Q  is  easily  shown  to  equal 

M  -1  _ 

(I-Q)(I-Q)  ,  so  that  the  measure  p.^(M)  also  depends  on  the  funda¬ 

mental  matrix  T  =  (I  -  Q)  \  Specifically 


p  (M)  =  1 
if 


element 


of 


.  (5.34) 


The  matrix  (I  —  Q  )(I  —  Q)  has  elements  that  represent  the  mean  number 

of  entries  into  state  j  in  the  first  M-l  transitions;  that  is,  the  mean 

,  -1 

number  of  entries  matrix  (I  -  Q)  corrected  for  truncation  at  M. 


H.  Summary  of  Effectiveness  Measures  Derived 

In  summary,  the  submatrices  T  =  (I  -  Q)  1  and  A  =  (I  -  Q)  1R  found 
in  the  partitioned  form  of  (I  -  Pz)  [Eq.  (5.16)]  have  the  following 
uses  and  interpretation  when  the  process  starts  in  state  i  at  time  n  =  0: 

•  The  elements  a^k  of  A  =  (I  -  Q)  are  the  probabilities  of 
absorption  in  state  k,  as  measures  of  effectiveness,  they 
are  the  probabilities  of  mission  success.  When  premultiplied 
by  (I  -  Qm) , the  elements  are  the  probabilities  of  absorption 
by  time  M;  the  (i,f)th  element  of  the  product  is  (1  -  proba¬ 
bility  of  the  mission  not  failing  by  time  M) . 

•  The  elements  t.  .  of  T  =  (I  -  Q)~^  are  the  mean  number  of 
entries  into  a  transient  state  j;  they  represent  the  mean 
time  spent  in  state  during  the  exercise.  The  sum 

r 

j=i 

is  the  mean  time  before  absorption;  operationally,  this  is 
the  mean  duration  of  the  exercise. 
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•  The  (i,k)th  element  of  the  product  TA  =  {(I  -  Q)-1(I  -  Q)-1R] — 
when  divided  by  the  corresponding  element  of  A — is  the 
mean  time  to  absorb  in  state  k,  given  absorption  in  state  k. 
Operationally,  these  times  are  the  mean  time  until  mission 
success  (or  failure)  occurs,  given  that  success  (or  failure) 
occurs. 


I .  A  Four-State  Example 

Four  states,  the  allowed  transitions  between  these  states,  and  the 
values  of  the  transition  probabilities  p  are  shown  in  Figure  5.2. 

ij 


FIGURE  5.2  TRANSITION  DIAGRAM  WITH  TRANSITION  PROBABILITIES 


The  transition  matrix  is 


fl 

5 

1 

_l\ 

4 

8 

16 

16  1 

3 

1 

3 

8 

2 

32 

32 

0 

0 

1 

0 

0 

0 

0 

1 

(5.37) 


where  the  submatrices  Q  and  R  are 
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(5.38) 


The  fundamental  matrix  ( I  -  Q)  =  T  is  found  by  first  calculating 
(I  -  Q). 


(I  -  Q) 


Inverting  this  matrix  results  in 


(5.39) 


(5.40) 


Therefore,  if  the  process  starts  in  the  search  state  the  average 
number  of  entries  into  the  search  state  (before  absorption)  is  32/9  and 
the  average  number  of  entries  into  the  prosecute  false  contact  state  is 
40/9.  Since  these  are  also  the  average  lengths  of  time  spent  in  the 
respective  states,  the  process  spends  more  time  prosecuting  false  con¬ 
tacts  than  it  does  in  searching.  The  sum  of  the  elements  in  the  first 
row  (32/9  +  40/9  =  72/9  =  8)  is  the  mean  time  until  absorption,  or  the 
mean  duration  of  the  exercise  given  that  the  exercise  started  in  the 
search  state.  The  absorption  probability  matrix  (I  -  Q)  =  A  is 
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When  starting  in  search,  the  probability  of  detection  is  13/36  and 
the  probability  of  failing  to  detect  is  23/16  since  the  second  column 
represents  the  absorption  state  4  (detect),  and  the  first  row  is  asso¬ 
ciated  with  starting  in  state  1  (search). 


Conditional  mean  times  to  fail  and  detect  are  obtained  from  computing 

„  -1  -1 

(I  -  Q)  (I  -  Q)  R  =  TA  and  normalizing  by  dividing  by  the  appropriate 
absorption  probability.  The  result  is  the  matrix 


[e( i,  k) ] 


/1696 

X 

36 

/  324 

X 

23 

l  1704 

X 

3 

\  324 

X 

2 

896 

X 

36\ 

324 

X 

~A 

888 

X 

3 

324  / 


(8.193 
7.889 


7.658 

8.222 


(5.42) 


Reading  from  the  first  row  to  reflect  starting  in  search  (state  1) 
results  in 


e(l,3)  =  mean  time  to  fail  given  that  failure 


ultimately  occurs  =  8.193 


e(l,4)  =  mean  time  to  detect  given  that  detection 


(5.43) 


ultimately  occurs  =  7.658 


As  a  check,  one  may  verify  that  the  weighted  sum  of  these  conditional 
mean  times — where  the  weights  are  the  corresponding  absorption  probabili¬ 
ties — equals  the  mean  duration  of  the  exercise. 


The  probabilities  of  fail  and  success  at  time  M  are  more  difficult 

th 

to  obtain  numerically  because  the  matrix  Q  must  be  raised  to  the  M 

4 

power.  Choosing  M  =  4  arbitrarily  and  calculating  Q  gives 


M  4 
Q  =  Q 


0.220 

0.366 

0,220 

0,366 
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(5.44) 


The  probability  of  detect  by  time  M  =  4  is  therefore  0.160  and  the  proba¬ 
bility  of  fail  by  that  time  is  0.254, 

The  imbedded  chain  corresponding  to  Figure  5.2  is  shown  in  Figure 
5.3  and  was  obtained  from  Figure  5.2  by  removing  the  self- transitions  at 
states  1  and  2  and  normalizing  the  remaining  transition  probabilities  to 


FAILURE  TO  DETECT 


DETECTION  OF  SUBMARINE 


FIGURE  5.3  IMBEDDED  CHAIN  FOR  FIGURE  5.2 


other  states.  Transition  probabilities  py  now  have  a  different  meaning. 

ij 


They  are  conditioned  on  leaving  state  i: 


p'  .  =  P[S(n  +  1) 
ij 


=  j  S(n)  =  i  and  S(n  +  1)  ^  i] 


for  i  =  1,  2}  and  all  j 


The  transition  probability  matrix  for  the  imbedded  chain  P'  is 
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(5.46) 


(5.47) 


so  that  the  mean  number  of  entries  into  states  1  and  2  are  24/9  and  20/9, 
respectively.  In  terms  of  the  original  chain,  24/9  is  the  mean  number  of 
searches  before  absorption  and  the  mean  number  of  false  contact  prosecu¬ 
tions,  respectively.  Since  the  average  time  in  the  search  state  per 
entry  into  search  is  1/(1  -  1/4)  =  4/3,  the  average  time  in  search  before 
absorption  is 

24  4  32 

(mean  number  of  searches)  X  (mean  length  of  one  search)  =  —  X  —  =:  — 


which  agrees  with  the  mean  time  in  search  calculation  performed  above  in 
the  original  chain. 

It  can  be  easily  verified  that  absorption  probabilities  are  the  same 
for  the  imbedded  chain  as  for  the  original  chain,  i.e., 

Af  -  (I  -  Q ' ) ~ 1R '  =  (I  -  Q) _1R  =  A  .  (5.48) 
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VI  PARAMETER  ESTIMATION  FOR  MARKOV  PROCESSES 


A.  Introduction 

Section  V  outlined  Markov  chain  theory  and  derived  matrix  formulas 
for  calculating  three  measures  of  effectiveness:  P(success);  mean  time 
until  mission  failure,  given  that  failure  occurs;  and  P(fail  does  not 
occur  by  a  specified  time  M) .  All  of  these  measures  were  calculated  in 
terms  of  the  transition  probabilities  p 

ij 

In  analytical  models  such  as  those  in  Ref.  6,  the  transition  proba¬ 
bilities  are  found  by  analytical  means.  However,  for  reasons  indicated 
in  Section  III-A,  operational  models  are  of  more  interest  in  this  report 
and  require  statistical  methods  to  estimate  parameters  from  operational 
data.  Several  methods  for  estimating  parameters  are  developed  in  this 
section,  assuming  that  the  state  space  is  given.  Least  squares  and  qua¬ 
dratic  programming  methods  are  shown  to  be  applicable  in  many  situations. 
Some  of  the  parameter  estimation  methods  are  applied  to  artificial  prob¬ 
lems  and  to  simulation  in  Sections  IX,  XI,  and  XII. 

Weighting  methods  for  combining  two  or  more  estimates  of  a  parameter 
are  treated  in  Section  XIII. 

B.  Factoring  the  Estimation  Problem 

In  some  applications,  factoring  the  parameter  estimation  into  two 
problems  is  worth  consideration.  The  first  problem  is  to  estimate  the 
parameters  of  the  imbedded  Markov  chain,  where  this  chain  is  found  from 
a  given  chain  by  removing  all  self-transitions  and  normalizing.  Alge¬ 
braically,  the  p  are  set  to  zero  for  all  transient  states  i,  and  the 
other  p  in  the  same  row  are  divided  by  1  -  p  so  that  the  sum  of  the 

ij  ii 
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outgoing  probabilities  is  unity: 


V  ij  1  /, 

L-J  1  -  p. .  1  -  P. .  \  ii 

i?j  11  11 

The  imbedded  chain  for  the  four-state  example  of  Figure  5.2  was  shown  in 
Figure  5.3  of  Section  V. 

Note  that  transitions  before  absorption  in  the  imbedded  chain  are 

always  to  a  different  state.  In  this  chain,  the  amount  of  time  spent  in 

a  transient  state  is  not  properly  accounted  for;  hence,  the  correspondence 

between  state  and  time  is  lost.  In  particular,  P(state)  as  a  function 

of  time  and  such  quantities  as  mean  times  till  absorption  are  generally 
( 

different  from  the  corresponding  quantities  in  the  original  chain.  How¬ 
ever,  since  transitions  out  of  a  state  occur  to  other  states  in  the  same 
ratio  as  in  the  original  process,  the  absorption  probabilities  are  iden¬ 
tical  to  those  in  the  original  process. 

Inasmuch  as  the  time  relationship  is  destroyed  in  the  imbedded  chain, 
it  is  considered  static  in  the  same  sense  that  a  conditional  probability 
model  is  static.  Putting  the  self-transitions  back  in  makes  the  model 
dynamic.  Fortunately,  the  self- transitions  can  be  added  to  the  imbedded 
chain  without  disturbing  the  static  (transition)  structure,  which  implies 
that  the  static  and  dynamic  aspects  of  the  model  can  be  studied  and 
analyzed  somewhat  independently.  This  separation  of  estimation  into  two 
parts  (imbedded  chain  and  holding  time)  is  used  in  later  examples  and 
is  explained  in  Subsection  D. 


j  =  1  for  all  transient  states  i 


C.  Estimating  Transition  Probabilities 


1.  Using  Transition  Data  (n^j) 


In  a  true  Markov  context,  the  most  desirable  data  for  estimating 
transition  probabilities  are  counts  of  the  number  of  transitions  ( n _ ) 
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from  state  i  to  state  j  for  all  states  i  and  j.  Transition  counts  may 
be  available  for  either  the  original  or  the  imbedded  Markov  chain.  In 


the  original  chain,  at  least  some  of  the  n. .  will  be  nonzero;  but  in  the 
imbedded  chain,  all  n  are  zero  by  definition.  When  the  n  for  the  im- 

ii  ij 

bedded  chain  are  given,  the  parameters  p  that  are  found  will  be  for  the 

ij 

imbedded  chain;  other  data  must  be  used  to  estimate  the  p  and  to  nor- 

ii 

malize  the  p  by  multiplying  each  by  the  factor  p  .  Let 

ij  ii 


n  =  observed  number  of  transitions  from  state  i  to  state  j 

ij 


■i-2X 


=  number  of  transitions  out  of  state  i 


Then  the  quantity  p  is  the  optimal  estimator  for  p  in  both 

ij  ij 

the  maximum- likelihood  and  minimum  chi-squared  sense: 


Pij  = 


n  ,/n  for  all  i, j  when  n  >0 

ij  i  i 


otherwise 


Obviously,  each  p  lies  in  the  closed  interval  (0,1),  Also,  by  the 

ij 

definition  of  n.,  the  sum  over  j  for  a  fixed  i  is  always  unity. 


This  estimator  is  entirely  analogous  to  that  used  to  estimate 

the  parameter  p  in  a  series  of  Bernoulli  trials  because  it  equals  the 

ratio  of  the  number  of  successes  to  the  number  of  trials.  Furthermore, 

p  is  easily  shown  to  be  unbiased  (its  expected  value  is  p  )  and  to  be 
ij  ij 

the  minimum  variance  unbiased  estimator.7  With  these  properties,  it  is 
unlikely  that  this  estimator  can  be  improved  upon.  The  difficulty  with 
this  estimator  is  primarily  that  considerable  effort  may  be  required  to 
determine  the  transition  counts;  currently,  they  are  rarely  given  in 
ASW  exercise  reports. 
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Determination  of  the  n  from  an  ASW  exercise  is  simple  in 

ij 

principle.  From  the  beginning  of  the  exercise,  the  times  when  state 
changes  (transitions)  occur  are  recorded  with  the  corresponding  transi¬ 
tion  times.  Whenever  it  is  known  whether  the  process  is  Markov  at  the 
outset,  it  is  desirable  to  first  represent  the  data  in  path  form;  that 
is,  first  record  the  sequence  of  states  entered  from  start  until  absorp¬ 
tion,  together  with  the  times  of  transition.  A  sample  path  that  starts 
in  state  1  and  is  absorbed  in  state  5  may  be  represented  by  two  rows  of 
data — one  row  for  the  states  entered  and  another  for  the  transition  times. 
An  example  is  the  data 

state  sequence:  12123125 

transition  times:  0  5  6  8  9  11  12  14 

An  equivalent  path  form  that  can  be  used  when  transitions  can 
be  assumed  to  occur  at  only  integer  times  is  the  sequence 

111112112331225 

th 

Reading  from  the  left,  the  n  element  of  this  sequence  is  the  observed 
state  at  time  n,  for  n  -  0,  1,  ...,  14.  For  theoretical  purposes,  the 
second  representation  is  preferable  because  all  the  information  is  shown 
in  transition  form.  In  either  case,  the  n.  .  can  readily  be  determined 
from  the  path  data  by  simply  counting  transitions.  The  matrix  of  n.  . 
for  the  single  path  shown  above  is  shown  in  Table  6.1.  A  single  "one" 
is  shown  for  the  5-5  transition;  this  results  in  the  desired  estimate  of 
unity  for  state  5  that  is  assumed  absorbing. 

The  transition  counts  for  other  paths  may  be  accumulated  in 
the  same  n  matrix,  and  one  matrix  for  each  path  need  not  be  recorded. 

ij 

For  example,  if  the  data  consist  of  three  paths  shown  on  the  left  for  a 
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Table  6.1 


TRANSITION  COUNTS  n  FOR  A  SINGLE  PATH 

ij 


j  1 

2 

3 

4 

5 

n. 

i 

1 

5 

3 

0 

0 

0 

8 

2 

1 

1 

1 

0 

1 

4 

3 

1 

0 

1 

0 

0 

2 

4 

0 

0 

0 

0 

0 

(0) 

5 

0 

0 

0 

0 

1 

(1) 

five-state  process  with  absorbing  states  4  and  5,  the  matrix  is  that 
shown  on  the  right  in  Table  6.2. 


Table  6.2 

TRANSITION  COUNTS  n  FOR  THREE  PATHS 

ij 


When  experimentation  is  needed  to  find  a  satisfactory  set  of 
states,  it  may  be  desirable  to  proceed  differently.  First,  a  number  of 
variables  may  be  selected  from  which  states  are  to  be  defined;  these 
values  are  recorded  continuously  by  going  through  the  exercise  or  sequence 
of  tactical  events  one  time.  For  a  tentative  definition  of  state  space, 
one  can  proceed  through  the  list  of  these  variables  to  determine  when 
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changes  of  state  occur  and  to  record  the  transitions  accordingly.  When 

the  situation  is  complex,  having  a  computer  program  compute  the  n  from 

ij 

lists  of  the  auxiliary  variables  may  be  desirable.  Suggestions  about 
variables  to  choose  for  defining  states  are  made  in  Section  IV-D-1. 

2 .  Using  Mean  Time- in-State  Data  (tj-j) 

In  some  tactical  situations  in  which  state  is  defined,  the 
total  time  spent  in  each  state  is  tabulated  in  the  exercise  report.  An 
example  of  this  form  of  data  is  the  tabulation  of  the  total  time  the 
Orange  submarines  are  in  the  defined  states  for  the  recent  series  of 
UPTIDE  exercises.  If  the  underlying  process  is  Markovian  (that  is,  if 
these  states  are  actually  states  in  the  Markovian  sense)  these  data  may 
be  used  to  estimate  the  transition  probabilities  in  a  Markov  chain.  Be¬ 
cause  less  data  collection  and  data  analysis  effort  should  be  required 
to  obtain  the  mean  time-in-state  data,  developing  a  parameter  estimation 
method  for  these  data  is  worthwhile. 

At  least  two  theoretical  difficulties  must  be  faced,  however. 
First,  the  theory  requires  the  mean  time  data  to  be  available  for  all 
possible  starting  (transient)  states.  Second,  only  the  transition  proba¬ 
bilities  for  the  transient  states  can  be  determined  so  that  additional 
data  are  required  for  the  calculation  of  absorption  probabilities  and 
other  quantities  involving  transitions  into  absorbing  states. 

The  basis  for  the  estimators  is  the  theoretical  expression  given 
in  Section  V  that  relates  the  transient  portion  (Q)  of  the  transition 
matrix  (P)  to  the  mean  number  of  entries  into  each  transient  state  before 
absorption.  The  matrix  equation  given  there  was 

T  =  (I  -  Q)_1  (6.1) 
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where  the  elements  of  T,  denoted  by  t  ,  were  the  mean  numbers  of  transi- 

ij 

tions  into  transient  states  j  before  absorption,  conditioned  on  the 
process  starting  in  state  i. 

The  elements  of  Q  are  simply  p  for  transient  states  i  and  j. 

i  j 

When  estimates  of  all  the  t. .  are  given,  we  may  denote  the  resulting 
matrix  by  T,  with  entries  \  .  When  the  starting  state  is  always  state 

ij 

1  (as  it  is  assumed  to  be  in  most  of  this  report),  the  second  and  later 
rows  of  T  may  possibly  be  filled  in  by  using  the  same  data  used  in  the 
first  row  after  all  data  were  removed  before  the  first  entry  into  state 
i.  (This  method  was  used  successfully  on  an  example  reported  in  Section 
XII  which  may  be  consulted  for  details.) 

The  straightforward  approach  to  estimating  the  elements  of  Q 
in  Eq.  (6.1)  would  be  simply  to  invert  both  sides  and  transpose  elements 
and  thereby  obtain  a  matrix  Q  whose  elements  are  estimates  of  the  transi¬ 
tion  probabilities  p 

ij 

Q  =  I  -  (T)"1  .  (6.2) 

When  a  satisfactory  fit  is  obtained,  this  simple  method  may  be  adequate. 
However,  possible  difficulties  include: 

A 

•  T  may  not  have  an  inverse,  or  may  be  very  nearly 
singular. 

•  Some  elements  of  Q  may  be  negative,  or  greater  than 
unity. 

•  Nonzero  elements  may  result  for  some  transitions  known 
to  be  physically  impossible. 

A  procedure  that  circumvents  the  first  and  last  difficulty  may 
be  illustrated  by  an  example.  Suppose  the  T  matrix  is  given  for  a  process 
with  three  transient  states  (designated  1,  2,  3)  and  two  absorbing  states. 
On  physical  grounds,  let  us  rule  out  transitions  1-1  and  2  -  3  so  that 
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the  transient  part  of  the  transition  matrix  is  of  the  form 


(here  the  carets  (~ )  are  omitted  for  convenience).  Thus,  I  -  Q  becomes 


I  -  Q  = 


Equation  (6.1)  was 


(6.3) 


(6.4) 


Premultiplying  each  side  of  Eq.  (6.4)  by  ( I  -  Q)  in  Eq.  (6.3)  gives 

(I  -  Q)T  =  (I  -  Q) ( I  -  Q)"1  =  I  ,  (6.5) 


or  by  expansion  and  transposition 

QT  =  T  -  I  .  (6.6) 

« 

The  matrix  equation  in  Eq.  (6.6)  represents  a  set  of  nine 
simultaneous  linear  equations  in  the  seven  unknown  transition  probabilities 
for  the  allowed  transitions.  When  ( I  -  Q)  has  an  inverse,  solving  Eq. 

(6.6)  is  equivalent  to  solving  this  set  of  equations,  which  in  expanded 
form  are: 
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1 


t  p 
21*12 

+ 

t  p 
31*13 

t 

11 

t  p 
22*12 

+ 

t  p 
32*13 

= 

t12 

t  p 
23*12 

+ 

t  p 
33*13 

= 

t13 

t  p 

11  21 

+ 

t  p 

21  22 

= 

t21 

t  p 

12  21 

+ 

t  p 

22  22 

= 

t22 

t  p 
13*21 

+ 

t  p 

23  22 

= 

t23 

t 

11 

P31 

+ 

t  p 
21*32 

+ 

t  p 
31*33 

= 

t31 

t12 

P31 

+ 

t  p 

22  32 

+ 

t  p 

32  33 

= 

t32 

t13 

P31 

+ 

t  p 

23  32 

+ 

t  p 

33  33 

= 

t33 

Eq. 

(6. 

7) 

may  or 

may  not  have  a 

-  1 


-  1 


(6.7) 


braic  sense.  However,  a  least  squares  solution  can  always  be  found. 
Equation  (6.7)  can  be  written  in  the  form 


Ap  =  b 


(6.8) 


where  A  is  a  9  X  7  matrix,  p  is  a  7  X  1  matrix  (i.e.,  a  column  vector 

with  7  components),  and  b  is  a  9  X  1  matrix.  Specifically,  p  may  be 

T 

chosen  as  the  vector  (p  p  P  ,  p  ,  p„. ,  p„  .  p  )  ,  where  T  de- 

12'  13'  21'  22'  31'  32'  33  ' 

notes  transpose,  and  b  may  be  chosen  as  the  column  of  values  on  the  right 
Side  of  Eq.  (6.7).  The  first  row  of  A  is  then  (t  ,  t  ,  0,  ® 3  0 3  0) > 

the  zeroes  reflecting  the  absence  of  all  p.  .  excepting  p^  and  in  the 

first  equation  of  Eq.  (6.7). 

The  modern  approach  to  least  squares  proceeds  directly  from  the 
matrix  form  in  Eq.  (6.8).  The  sum  of  the  squared  deviations  is 
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(6.9) 


2  T  /  T  T  T\ 

d  =  (Ap  -  b)  (Ap  -  b)  =  Ip  A  -  b  y(Ap  -  b) 

RT  TT  T  T  TT  T  T 

=pAAp-pAb-bAp+bb=pAAp-2bAp+bb 

2 

The  derivative  of  the  scalar  d  with  respect  to  the  vector  p  is  then 
found  and  set  to  zero: 


T  T 

2A  Ap  -  2A  b  =  0 


(6.10) 


Solving  this  by  inverting  yields  the  unconstrained  least-squares  solution 


(6.11) 


Nothing  in  this  formulation  constrains  the  unknowns 
satisfy  the  probability  constraints 


to 


p  >  0 

ij 


(6.12) 


If  these  constraints  are  badly  violated,  a  quadratic  programming  formula¬ 
tion  may  be  used.  Appendix  D  explains  quadratic  programming  in  more  de¬ 
tail,  but  here  it  need  only  be  said  that  arbitrary  linear  inequality  and' 
equality  constraints  may  be  introduced;  and  in  particular,  the  probability 
constraints  Eq.  (6.12)  can  be  added.  The  general  form  of  a  quadratic 
program  is:  find  a  vector  x  to  minimize 


T 

z  =  x  Dx 


(6.12) 


subject  to  constraints 
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A  x 

1 


A  x  £  b 
2 


2 


A  x  >  b 
3 


3 


x  >  0 


(6.13) 


where  D,  A  ,  A  ,  A  are  given  matrices  of  constants:  the  b  ,  b  ,  b  are 
’  l’  2}  3  i;  2}  3 

given  column  vectors;  and  the  scalar  z  (the  sum  of  the  squared  deviations) 
is  to  be  minimized. 

3.  Using  Mean  First-Passage  Times  (m.^-) 

When  a  set  of  states  is  defined  for  a  class  of  tactical  situa¬ 
tions,  it  is  possible  to  obtain  transition  probability  estimates  from  data 
on  the  time  between  the  entry  into  one  state  and  the  entry  into  another. 

To  illustrate  this,  suppose  that  search,  detect,  and  classify  are  states 
in  a  Markov  chain  model.  Then  such  quantities  as  the  average  time  from 
the  beginning  of  search  until  detect  (i.e,.,  entry  into  the  detect  state), 
the  average  time  from  the  beginning  of  search  until  the  time  of  entry 
into  classification,  the  average  time  from  the  entry  of  detection  until 
the  entry  of  classification  can  be  estimated  from  operational  data.  These 
quantities  may  themselves  be  considered  as  dynamic  measures  of  effective¬ 
ness  since  they  depend  upon  the  dynamics  of  the  ASW  system  or  unit  being 
considered.  For  the  current  purpose,  however,  they  are  not  considered  as 
end-products  but  as  parameters  required  for  the  estimation  of  the  four 
measures  of  effectiveness  considered  earlier.  All  of  these  (end-product) 
measures  of  effectiveness  are  calculated  from  the  transition  probabilities 
in  a  Markov  chain  model  so  methods  are  necessary  for  estimating  the  transi¬ 
tion  probabilities  from  time-be tween-states  information. 


89 


In  technical  terms,  a  set  of  quantities  of  interest  from  which 
transition  probabilities  may  be  estimated  are  called  first-passage  times. 
The  first-passage  time  from  state  i  to  state  j  is  defined  as  the  number 
of  transitions  to  arrive  in  state  j,  given  that  the  state  is  i  at  time 
zero.  First-passage  times  are  defined  only  for  recurrent  chains;  that 
is,  for  Markov  chains  all  of  whose  states  can  be  reached  from  any  other 
state.  When  all  states  do  not  communicate  in  this  manner,  a  modified 
definition  entailing  conditional  first-passage  times  may  be  used;  but  this 
results  in  very  complex  expressions.  Therefore,  the  formulation  given 
here  is  for  the  case  all  states  communicate;  by  definition,  there  are  no 
absorbing  states.  The  measure  of  effectiveness  appropriate  to  these 
situations  is  the  probability  of  occupying  a  designated  state  at  a  ran¬ 
domly  chosen  time;  this  is  appropriate,  for  example,  for  the  trailing 
mission. 

Data  requirements  are  a  complete  set  of  estimates  of  mean  first 
passage  times  (m.  .),  where  "complete"  means  all  (i,j)  combinations  that 
are  physically  and  logically  possible.  As  in  the  previous  section,  using 
data  from  paths  that  all  start  in  the  same  state  may  be  possible  by  ig¬ 
noring  all  transitions  before  the  first  entry  into  state  i  when  data  for 
row  i  are  being  calculated. 

A  possible  objection  to  the  use  of  mean  first  passage  time  data 
may  be  raised  at  this  point.  When  the  analysis  required  to  obtain  the 
mean  first-passage  times  is  considered,  it  seems  that  paths  must  be  traced 
from  each  possible  starting  state  (i)  to  the  entry  of  an  absorbing  state 
(k),  and  that  this  tracing  would  yield  path  information  that  can  be  trans¬ 
lated  into  transition  counts  n  .  Since  methods  are  available  for  esti- 

ij 

mating  the  transition  probabilities  p  from  the  n  ,  it  then  might  be 

ij  ij 

asked  whether  the  mean  first-passage  times  contribute  anything  new  to  the 
problem.  The  answer  to  this  will  depend  on  whether  the  process  is  actually 
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Markovian,  the  measure  of  effectiveness,  and  what  the  "goodness  of  fit" 
measure  is  for  the  model.  In  a  true  Markov  context  where  only  the  ab¬ 
sorption  probabilities  are  desired,  the  transition  information  is  suffi¬ 
cient.  However,  if  the  model  is  realistically  at  least  somewhat  non- 
Markovian  and  is  also  required  to  fit  dynamically  [in  the  P(state)  as 
a  function  of  time  sense]  then  the  transition  probabilities  derived  from 
the  mean  first-passage  times  should  give  useful  additional  information. 


Equations  relating  the  mean  first  passage  times  and  transition 
probabilities  may  be  found  in  any  standard  text  on  Markov  chains.8 
Letting  m  be  the  mean  first  passage  time  from  i  to  j  and  p  be  the 

ij  ij 

usual  transition  probability  from  i  to  j,  the  equations  are 


m 

ij 


1  + 


k=l 


p  m 

ij  kj 


1  ^  i,  j  £  N 


(6. 14) 


k^j 

2  2 

This  is  a  set  of  N  linear  equations  in  the  N  unknown  transition  proba¬ 
bilities.  In  theory,  with  exact  values  of  the  m.^  known,  there  should  be 
a  unique  solution  to  this  linear  system  that  yields  the  transition  proba¬ 
bilities.  The  probabilities  comprising  the  solution  should  automatically 
satisfy  the  nonnegativity  and  row  sum  conditions 


N 

p  >  0  ,  /  p  .  =  1  for  i  =  1,  2,  ...,  N  .  (6.15) 

ij  IJ 

j=l 

In  applications,  however,  there  are  two  reasons  why  straight¬ 
forward  solution  of  the  set  of  linear  equations  in  Eq.  (6.14)  will  seldom 
be  possible.  First,  the  model  will  rarely  be  exactly  Markovian  as  re¬ 
quired  by  the  theory.  Second,  the  amount  of  data  is  limited  so  that  the 
estimators  m  will  have  noise  components  even  in  the  Markovian  case. 

ij 

Three  outcomes  are  possible  when  one  attempts  to  solve  the  system  in  Eq. 
(6.14). 
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(1) 


The  system  of  equations  in  Eq 
conditions  in  Eq.  (6.15)"1 


(6.14)  may  have  a  solu¬ 
tion  with  nonnegative  p^ .  which  also  satisfies  the  row 


(2)  The  system  may  have  a  solution  with  some  p^j  outside 
the  range  zero  to  unity,  or  the  row  conditions  in  Eq. 
(6.15)  may  not  be  satisfied. 

(3)  The  system  may  be  inconsistent  and  hence  have  no 
solution. 


The  last  two  outcomes  are  the  result  of  the  non-Markovian  of  the  real 
processor  noise  components  in  the  estimators.  Only  the  first  outcome 
gives  an  acceptable  solution,  but  the  second  and  third  possibilities  are 
often  to  be  expected  in  working  with  a  real  problem. 


a.  Constrained  Least  Squares 

A  first  method  for  obtaining  an  acceptable  set  of  estimates 
from  the  set  of  equations  of  Eq.  (6.14)  will  be  through  the  application 
of  constrained  least  squares.  This  model  is  ordinary  least  squares  aug¬ 
mented  by  linear  constraints  on  the  unknown  quantities.  The  constrained 
least  squares  formulation  follows.  The  solution  always  exists,  but  the 
nonnegativity  constraint  may  be  violated. 

First,  the  system  in  Eq.  (6.14)  is  replaced  by  a  single 
quantity  to  be  minimized: 


2 


d 


N  N_ 

EE 


i=l  j=l 


p  m 
ik  kj 


(6.16) 


2 

The  quantity  d  is  obviously  nonnegative  and  the  lowest  possible  value  is 
therefore  zero  in  Eq.  (6.16).  The  value  zero  is  achieved  as  a  minimum 
if — and  only  if — the  set  of  equations  in  Eq.  (6.14)  is  consistent.  To 
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ensure  acceptability  of  the  minimizing  set  of  p.  .,  however,  it  is  neces¬ 
sary  to  add  the  row  constraints  of  Eq.  (6.15).  The  constrained  least- 
squares  problem  then  takes  the  form  of  minimizing 


2 

d  by 


selecting  p 

ij 


(i,j  =  1,  2, 


N) 


(6.17) 


to  satisfy 


N 


for  i  =  1,  2,  3,  .  .  .  ,  N 


(6.18) 


The  double  summation  defining  d  can  be  rewritten  in  matrix  form  as 


2  T 

d  =  (Ap  -  b)  (Ap  -  b) 


(6.19) 


when  b  is  an  observable  random  vector,  p  is  the  unknown  vector  of  tran- 

ij 

sition  probabilities,  and  A  is  a  matrix  of  constants  determined  from  the 
m  .  The  row  constraints  of  Eq.  (6.19)  also  can  be  written  in  matrix 

ij 

form  as 


Bp  =  k 


(6.20) 


where  k  is  a  known  vector  and  B  a  known  matrix.  The  solution  to  Eq. 
(6.17)  subject  to  Eq.  (6.18)  is  then  that  given  in  Appendix  D  with  the 
covariance  matrix  V  equal  to  the  identity  matrix 


P 


A  i 

=  P  + 


(6.21) 


where  p  is  the  solution  to  the  unconstrained  pro-blem 


P  = 


(6.22) 
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b. 


Quadratic  Programming  Solution 


The  constrained  least  squares  formulation  minimized  a  sum 
of  squares  subject  to  linear  constraints  derived  from  the  row  constraints 
that  the  sum  of  the  transition  probabilities  out  of  each  state  should  sum 
to  unity.  Unfortunately,  there  is  a  possibility  that  some  probabilities 
may  lie  outside  the  range  zero  to  unity  while  satisfying  the  row  condi¬ 
tions.  When  the  nonnegativity  constraints 

p  .  >  0  for  i ,  j  =  1,  2,  3,  ...,  N  (6.28) 

ij 

are  also  introduced,  the  result  is  a  quadratic  program  to  minimize 

T  T 

p  Dp  +  c  p 


subject  to 


p  s  0 


Ap  =  b 


(6.24) 


where 

p  is  the  unknown  vector  of  transition  probabilities 
b  and  c  are  known  constant  vectors 

D  is  a  known  matrix  of  constants  which  is  symmetric  and  positive 
semidef inite . 


4.  Using  the  First  and  Second  Moments  of  First-Passage  Times 

In  addition  to  the  mean  first-passage  times  (the  first  moments), 

2 

estimates  of  the  second  moments  of  first-passage  time  (s  )  may  also  be 
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available.  For  a  Markov  chain,  the  following  relations  hold  when  all 
states  communicate.5 


2 

s  =  2m 

ij  ij 


k=i 


P.  s  . 

ik  ij 


i >  J  -  1,  2, 


N 


(6.25) 


This  set  of  equations  has  nearly  the  same  form  as  Eq.  (6.14)  in  the  pre¬ 
vious  section,  and  the  same  remarks  apply  concerning  its  solution.  In 
particular,  a  constrained  least- squares  approach  may  be  taken  to  obtain 

the  estimators  p  when  the  estimates  of  the  first  and  second  moments 
_2  iJ 

m  and  s  are  available.  Since  second  moments  are  used  as  well  as  first 

ij  ij 

moments  in  the  fitting  process,  the  statistical  properties  of  the  esti¬ 
mators  can  be  expected  to  be  at  least  somewhat  improved. 


5.  Using  P(State)  as  a  Function  of  Time 


Other  data  that  may  be  available  in  ASW  applications  are  esti¬ 
mates  of  the  probability  of  occupying  each  state  as  a  function  of  time. 
For  a  defined  set  of  states  with  time  origin  at  the  beginning  of  the 
engagement  or  exercise,  the  estimates  of  P(state)  for  a  fixed  t  may  be 
from  the  ratios 


p[S(t)  -  i] 


Number  of  times  state  i  was 

,  occupied  at  time  t 

y  (t)  =  - - - 

i  Number  of  times  state  i  could  have 

been  occupied  at  time  t 


(6.26) 


for  i  =  1,  2,  . . . ,  N.  That  is,  y.(t)  is  proportional  to  the  observed 
number  of  times  that  state  i  was  occupied  at  time  t,  the  denominator 
being  the  normalizing  constant  that  makes  the  sum  of  the  y.(t)  equal 
unity  for  each  fixed  t.  Other  ways  of  obtaining  the  estimates  y^(t)  may 
be  found  more  practical  in  some  applications.  In  economics  applications, 


the  y  (t)  are  often  from  such  aggregated  quantities  as  the  relative  pro- 
i 

portions  of  total  sales  for  each  competing  firm  in  a  particular  market. 
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Analogous  quantities  may  be  found  in  ASW,  for  example,  by  considering  the 
relative  proportions  of  time  a  submarine  spends  in  various  information 
states.  Because  deceptive  tactics  and  decoys  are  used  to  force  a  sub¬ 
marine  into  lower  information  states,  this  may  be  a  way  to  study  the  im¬ 
pact  of  particular  tactics/decoy s  on  transition  probabilities  and  hence 
on  measures  of  effectiveness  derived  from  these  transition  probabilities. 

To  proceed  with  the  mathematical  development,  let  the  N  state 
process  be  observed  from  time  t  =  0  until  time  t  =  T.  Let  the  given  data 
be 


y .( t)  =  p[S( t)  =  j] 
J 


T 

N 


(6.27) 


Then  the  relationship  between  the  unknown  transition  probabilities  p 

ij 

and  the  observed  y.(t)  are 


N 

yj(t)  =  ^2  yi(t  “  1)p  —  +  Uj(t)  j  =  ]’  l  .  •  (6.28) 


i=l 


ij 


t  —  1,  2,  ...,  T 


If  the  y.(t)  were  the  P[S(t)  =  j]  from  a  true  Markov  process 
with  parameters  p.  .,  all  the  error  terms  u.(t)  would  be  zero.  The  approach 
will  again  be  to  apply  least  squares;  a  set  of  p  is  sought  to  minimize 

ij 

the  sum  of  the  squared  errors. 

Vector  and  matrix  notation  is  needed  for  compactness.  Accord¬ 


ingly,  let 


y.  =  ry.U),  y.(2),  ...,  y.(T)~]T 

j  L  j  j  j  -1 

pj  *  (v  p2j’  •••’  PNj)T 

u.  =  fu.(l),  u  .  ( 2)  ,  .  ..,  u.(T)~]T 
J  L  j  j  j  -I 


(6.29) 

(6.30) 

(6.31) 
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y1(0) 

y2(0) 

. . . 

y  (0) 

J'n 

A  = 
j 

Vt- 

i) 

y2(t  - 

i) 

•  •  • 

y  (t  - 

■’n 

1) 

(6.32) 

yx(t  - 

i) 

y2(T  - 

i) 

•  •  • 

y  (T  - 

1) 

(note  that  A  is  actually  independent  of  j).  Then  in  partitioned  (com¬ 
ponent)  form,  the  Eqs.  (6.28)  become 


y. 

-  — 

A 

p 

u. 

i 

1 

1 

1 

y 

A 

+ 

u 

2 

2 

2 

• 

• 

• 

* 

• 

• 

* 

1 - 

< 

_ 1 

^PN_ 

_UN_ 

Equations  (6.33)  can  be  written  more  compactly  as 


(6.33) 


y  =  Ap  +  u 


(6.34) 


Conventional  least  squares  can  be  applied:  minimize  the  sum  of  the 

T 

squared  deviations  u  u  by  minimizing 


T  T 

u  u  =  (y  -  Ap)  (y  -  Ap) 


(6.35) 


whose  solution  is  the  optimal  estimator  p 


-  (A)'1, 


T 

A  y 


(6.36) 


provied  that  A  A  is  nonsingular  (which  it  is  when  A  is  of  full  rank). 
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It  can  be  shown  that  the  row  sum  conditions  (E  p  =1  for  all 

j 

i)  are  automatically  satisfied  by  p,  but  the  p  are  not  guaranteed  to 

ij 

lie  between  zero  and  unity.9 

Estimates  that  restrict  the  transition  probabilities  to  the 
range  (0,1)  may  also  be  developed.  One  again  minimizes  the  scalar 
quantity 

d2  =  (y  -  Ap)T(y  -  Ap)  ,  (6.37) 

subject  to  the  row  constraints  and  p  >  0.  This  is  again  a  quadratic 
program,  soluble  with  a  modified  simplex  algorithm. 

The  least-squares  model  is  applied  to  the  y  (t)  calculated  from 
an  artificially  constructed  non-Markov  process  in  Section  XI. 

D.  Estimating  the  Density  Function  of  the  Transition  Times 
1.  Introduction 

In  a  Markov  chain,  the  only  parameters  are  the  transition 
probabilities  p  ;  these  determine  the  static  (transition)  and  the 

ij 

dynamic  behavior  of  the  process.  As  mentioned  in  Subsection  B,  the 

diagonal  terms  (p  )  determine  the  holding  (or  waiting)  times  in  the 
ii 

states  and  hence  determine  the  dynamic  behavior.  It  is  often  convenient 
to  estimate  the  p  after  the  off-diagonal  p  is  estimated  by  data  from 

ii  ij 

the  imbedded  chain — that  is,  from  data  that  depended  only  on  transitions 
made  between  different  states. 

This  section  begins  by  showing  how  to  estimate  the  p  when  the 
other  p  and  average-time-in-state  data  (T  )  are  available.  The  method 

ij  i 

can  be  applied  when  it  can  be  assumed  that  the  holding  time  in  a  state  is 
geometrically  distributed  and  is  independent  of  the  state  to  which  the 
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transition  is  made.  Since  these  conditions  are  often  not  satisfied  in 
applications,  methods  are  given  that  may  be  tried  when  the  dynamic  fit 
obtained  by  using  a  Markov  chain  is  unsatisfactory.  Techniques  of  this 
kind  generally  use  auxiliary  states  to  represent  the  transition  time  be¬ 
tween  selected  pairs  of  states  (i,j).  When  the  density  function  of  the 
transition  times  is  chosen  to  be  the  functional  form  h  (n),  the  model 

ij 

is  called  a  semi-Markov  model;  some  techniques  for  estimating  the 
parameters  of  these  densities  conclude  the  section. 


2.  The  Holding  Time  Parameter  (p^) 

In  the  discrete  time  Markov  process  (Markov  chain),  the  transi¬ 
tion  probabilities  determine  both  the  static  (transition)  and  dynamic  be¬ 
havior  of  the  process.  In  particular,  the  density  function  of  the  time 
spent  in  a  state  is  geometrically  distributed,  independent  of  the  destina¬ 
tion  state.  Denoting  the  self- transition  probability  of  state  i  by  p.., 
the  probability  of  making  a  transition  to  some  other  state  at  time  n  is 

p11  ^(1  -  p  ),  because  (n  -  1)  self- transitions  (returns  to  state  i)  must 
ii  ii 

first  occur,  followed  by  a  transition  to  some  other  state.  Therefore, 
the  average  number  of  transitions  required  to  leave  state  i  is 


T 

i 


■E 

n  =  1 


nP( state  i  is  left  on  the  n^*1 


transition  after  entry) 


=2 n  pii1(i  •  piJ  *  i1  -  pn) 


n=i 


(‘--J*  1 


-  P.  . 

ii 


(6.38) 


The  simplest  possible  estimation  procedure  to  fit  Markov  chain 
parameters  to  observed  data  on  transitions  in  the  imbedded  Markov  chain 
and  to  observed  average  transition  time  data  exploits  this  formula.  When 
the  mean  time  (T, )  spent  in  each  state  is  known  or  is  estimated  from 


operational  data,  the  optimal  estimator  p  is  found  by  solving  Eq.  (6.39) 

ii 
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(6.39) 


for  the  estimator  p  .  The  result  is 

ii 

p  =  1  -  1/T  .  (6.40) 

ii  i 

After  these  estimates  of  the  self- transitions  (the  diagonal  elements  of 
the  transition  matrix  P)  have  been  determined,  the  estimates  p/  for  the 

ij 

other  (off-diagonal)  transition  probabilities  for  the  imbedded  process 

can  be  normalized  by  multiplying  each  p/  by  (1  -  p  ).  The  resulting 

ij  ii 

set  of  p.  .  (i,j  =  1,  2,  . . . ,  N)  are  nonnegative,  satisfy  the  row  con¬ 
straints  for  each  i,  and  are  therefore  feasible  estimates  of  the  transi¬ 
tion  probabilities.  This  estimation  procedure  is  used  on  sample  problems 
in  Sections  IX  and  XI.  Based  on  these  examples,  when  the  actual  model  is 
non-Markovian  and  this  simple  procedure  is  used  without  further  modifica¬ 
tions,  a  poor  dynamic  fit  seems  to  be  expected. 

3.  Adding  Auxiliary  Delay  States 

One  possible  way  to  improve  the  dynamic  [P(state)  as  a  function 
of  time]  fit  entails  the  addition  of  auxiliary  states.  This  addition 
does  not  influence  the  probabilities  of  transitions  between  the  original 
states,  but  does  influence  the  times  from  one  state  to  another. 

If  it  is  reasonable  to  assume  that  the  waiting  time  density  is 
independent  of  the  destination  state,  the  natural  approach  is  to  increase 
the  number  of  parameters  that  characterize  the  waiting  time  density.  The 
obvious  two  parameters  are  the  mean  and  variance  of  the  waiting  time.  (A 
Markov  chain  waiting  time  density  has  only  a  single  parameter  (p  )  that 
determines  both  the  mean  waiting  time  and  the  variance  of  the  waiting 
time.)  An  approach  that  sometimes  works  is  to  add  auxiliary  delay  states, 
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say  D.  auxiliary  states  for  state  i,  to  give  the  resulting  waiting  time 

density  function  the  same  sample  mean  and  variance  as  the  estimated  waiting 

time  density.  If  m  is  the  (sample)  mean  waiting  time  in  state  i  and  v 

i  i 

is  the  corresponding  variance,  then  the  D.  should  be  chosen  to  satisfy 


D.  +  - — 

1  Pii 


=  m 


D  =  nonnegative  integer 
i 


(6.41) 


11 


MJ2  1 


(6.42) 


Equation  (6.41)  was  derived  to  fit  the  mean  waiting  time  by 

assuming  that  D  delay  states  (with  a  mean  time  of  delay  D  )  are  followed 
i  i 

by  the  original  state  i  [with  mean  time  l/(i  -  p  )].  Equation  (6.42), 

i  l 

derived  to  match  the  variances,  does  not  involve  D  since  the  delaying 

i 

transitions  always  occur  and  do  not  involve  the  variance.  Solving  Eq, 


(6.42)  for  p  ,  the  estimator  of  p  ,  gives 
ii  li 


P.  . 

ii 


_  (2  +  ±  +  Vj2 


(6.43) 


for  the  appropriate  choice  of  sign  before  the  radical.  Using  this  value 

of  p  in  Eq.  (6.41)  and  solving  for  the  number  of  delay  states  D  , 
ii  i 


D .  =  m  .  -  - - - - 

l  l  1  -  P 

ii 


(6.44) 


If  the  computed  value  of  D .  is  negative,  this  method  will  not  work.  In 

any  case,  rounding  to  an  integer  is  generally  necessary  and  results  in 

some  error  in  fit  in  the  mean  time.  The  fit  to  the  observed  mean  time 

is  usually  more  important  than  the  fit  to  the  observed  variance.  The 

rounded  D  may  now  be  used  to  determine  a  new  p  from  Eq.  (6.41).  Then 
i  ii 

the  error  in  variance  may  be  calculated  from  Eq,  (6.42);  if  the  error  is 
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not  excessive,  these  modified  estimates  of  p..  and  D.  may  be  used  in  the 

modified  Markov  chain  model  with  D  auxiliary  states. 

i 

A  numerical  comparison  between  the  original  simple  estimating 
method  and  the  auxiliary  delay  states  method  can  be  found  in  Section  IX. 
A  considerable  improvement  resulted  from  the  addition  of  delay  states  in 
this  example. 


4.  Adding  Auxiliary  States  and  Transitions 

The  delay  method  described  in  the  last  subsection  was  systematic, 
simple,  and  permitted  the  matching  of  two  parameters  to  an  observed  or 
theoretical  holding  time  density  function.  The  method  does  not  always 
work,  however,  and  in  any  case  the  auxiliary  delay  states  are  not  used 
efficiently.  This  inefficiency  results  from  the  simple  form  of  the  tran¬ 
sitions  among  the  delay  states,  for  each  delay  state  merely  "holds"  the 
process  in  state  i  for  one  more  transition.  When  the  fullest  possible 
set  of  transitions  are  allowed  between  the  auxiliary  states,  we  may  ex¬ 
pect  to  use  the  given  number  of  states  more  efficiently  by  selecting 
parameters  to  approximate  given  holding  time  densities.  This  is  the 
approach  taken  in  this  section. 

An  example  is  used  to  illustrate  the  approach.  Figure  6.1 
shows  an  example  in  which  three  auxiliary  states  (i7,  i." ,  iw)  and  the 
indicated  transitions  allowed  between  them  have  been  added  to  represent 
a  holding  time  density  h.(n),  where  h.(n)  =  P  (process  makes  a  transition 
out  of  state  i  at  time  n/state  i  was  entered  at  time  zero). 


For  definiteness,  suppose  that  all  the  conditional  transition 

probabilities  c  (conditioned  on  a  state  change  from  i  to  j)  are  given, 
ij 

These  c  are  determined  from  the  usual  transition  probabilities  p  ,  by 

ij 

normalizing  over  the  transitions  out  of  state  i 
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for  j  =  1,  2,  . .  . , 


>  •  •  •  i 


N 


c 

ij 


N 


k^i 


ik 


i  -  1,  i  +  1 


All  inputs  to  state  i  (the  original  state)  are  sent  to  state  i ' ,  and 

states  i"  and  x"  are  placed  between  i7  and  i  as  shown,  with  transition 

probabilities  for  all  auxiliary  branches  being  indicated  by  a  in  the 

i 

diagram.  The  overall  transition  structure  of  the  process  is  readily 
seen  to  be  unchanged  because  the  transitions  from  i  to  any  other  state  j 
occur  with  the  correct  probability  c  .  The  structure  of  the  network 

ij 

of  auxiliary  states  and  transitions  added  to  the  original  process  deter¬ 
mine  the  density  of  the  waiting  time. 


However, 


Ten  new  parameters  (a^  through  a  )  are  shown  on  the  diagram, 
the  linear  (row)  constraints 


a  +  a  +  a  =  1 
12  3 


a  +  a  +  a  =  1 
4  5  6 


a+a+a+a  =1 
7  8  9  10 


(6.45) 


reduce  the  number  of  free  parameters  to  seven.  By  allowing  these  ten 
parameters  to  vary  subject  to  the  three  row  constraints  and  the  usual 
nonnegativity  conditions  (a.  >0,  we  may  expect  to  realize  a  variety  of 
shapes  for  the  waiting  time  density.  Several  examples  of  densities  are 
shown  in  Figure  6.2  for  the  parameter  sets  detailed  on  the  figure. 

Analytically,  it  is  straightforward  to  determine  the  probability 

generating  function  g  (z)  of  the  waiting  time  density.  The  transmission 

i 

from  state  i;  to  state  i  is  g.(z). 

From  Mason's  formula  in  Appendix  C,  the  graph  determinant  A  of 
the  auxiliary  network  is 
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=  1  +  z(-a  -a  -  a  )  +  z  /a  a  +  a  a  +  aa  +  aa  +  a  a  'l 
\  1  4  7/  \  2  5  68  14  17  47/ 


+z  I -a  a  a„  -  a  a  a  -  a  a  a 
\  2  6  9  168  25 


)2  3 

=  1  +  Az  +  Bz  +  Cz  .  (6.46) 


ator  of  the  transmission  is 


-a  -  a„  -  a„ 
7  8  9 


Kl-a)z+aa(l-a 
4/  2  61  7 


a  -  a  z 
8  9/ 


PARAMETER  ARRAY  /  a1  a2  a3  0 
|  a5  a4  a6  0 


CASE  I  '*  1" 

1/3  1/3  1/3  0  .9  .05  .05  0  1/3  1/3  1/3  0 

1/3  1/3  1/3  0  - ►  .9  .05  .05  0 


1/3 

1/3 

1/3 

0 

1/3 

1/3 

1/3 

0 

1/4 

1/4 

1/4 

0 

0 

0 

0 

1 

CASE  IV 

.09  .  05  .05  0 

.05  .90  .05  0 


;025  .025  .7  .25  .7  .025  .025  .25 


Letting  A,  B,  and  C  be  the  constants  in  the  graph  determinant  as  in  Eq. 
(6.46),  the  transmission  g(z)  is  then  the  ratio 


g.  (z) 


Ta  (l  -  a  \z  +  a  a  z  ’""Ifl  -  a  -  a  -  a  "1 
L  3  \  4/  2  6  J  L  7  8  9J 

2  3 

1  +  Az  +  Bz  +  Cz 


(6.47) 


Expanding  g  (z)  in  an  infinite  series  in  the  variable  z  gives  the  density 
i 

n 

of  the  waiting  time  with  the  coefficient  of  z  equalling  the  probability 

st 

that  the  transition  out  of  i  occurs  on  the  (n  +  1)  step.  The  expansion 
results  in  complex  expressions  for  the  coefficients  and  therefore  does 
not  offer  a  constructive  method  for  finding  the  a.  parameters. 

A  straightforward  method  is  available  for  choosing  the  a  to 

i 

approximate  the  holding  time  density  h.(n).  If  the  cumulative  distribu¬ 
tion  of  the  waiting  time  is  denoted  by  H.(n) 


n 

H . (n)  =  ^  h.(m)  =  P  (holding  time  in  i  £  n)  ,  (6.48) 

m=o 


then  H.(n)  is  related  to  the  probability  of  being  absorbed  in  state 
4(=state  i)  of • the  auxiliary  network  by 


y^(n)  h  P  (absorption  in  state  4  by  time  n)  =  H.(n) 

for  n  =  0,  1,  2,  ...  .  (6.49) 

Therefore,  the  probability  of  occupying  one  of  the  states  (the  absorbing 

state  4)  is  given  as  a  function  of  time.  If  the  probabilities  of  occupying 

the  other  states  as  a  function  of  time  were  also  known,  the  least  squares 

approach  given  earlier  should  suffice.  To  apply  the  methods  of' Section 

VI-C,  estimates  of  the  P(state)  as  a  function  of  time  relations  [y^(n)] 

for  states  i  =  1,  2,  and  3  should  be  made  such  that  y  (n)  >  0  for 
4  1 

i  =  1,  2,  3  and  E  y  (n)  =  1  for  all  n.  The  form  of  these  y  ( n)  should 
i=l  i  i 
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not  be  important,  since  the  modification  to  the  least  squares  procedure 
will  weight  the  deviations  associated  with  the  first  three  y  (n)  much 
less  than  the  relation  of  interest  y^(n). 

From  estimates  of  the  y  (n)  over  a  range  t  =  0  to  T,  formulate 

i 

the  estimation  problem  for  the  a  can  be  formulated  as  the  minimization 

i 

problem 


2  T 

minimize  d  =  (Y  -  Aa)  W(y  -  Aa)  ,  (6.50) 

where  the  vector  a  replaces  to  the  vector  p  in  Eq.  (6.34)  and  the  weighting 
matrix  W  is  a  diagonal  matrix.  To  weight  the  y^(n)  deviations  heavily 
and  the  others  lightly,  a  matrix  W  such  as  that  shown  in  Eq.  (6.51)  may 
be  selected 


w  =  1 


3T  COLUMNS  T  COLUMNS 


(6.51) 


The  solution  for  the  optimal  vector  a  is  again  found  by  dif- 
2 

ferentiating  d  in  Eq.  (6.50)  with  respect  to  the  vector  a,  equating  the 
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result  to  zero,  and  solving  for  a  =  a: 


*  /  T  -1  \  T  -1 

a  =  (A  W  A  1  AW  y  .  (6.52) 

This  solution  has  the  same  potential  difficulties  as  earlier 

least  squares  procedures;  in  particular,  some  a.  may  be  outside  the 

interval  (0,1).  As  before,  Quadratic  programming  may  be  used  to  obtain 

feasible  a  . 

i 

The  auxiliary  state  method  may  be  used  in  many  other  situations 
to  approximate  holding  time  densities.  The  holding  time  density  may  be 
made  to  depend  on  the  terminal  state  by  adding  auxiliary  states  between 
state  i  and  state  j.  That  is,  auxiliary  states  and  transitions  are  added 
for  each  branch  of  the  original  transition  diagram  instead  of  for  each 
state. 

Several  destination  states  j  may  be  handled  simultaneously  by 

the  least  squares  approach  above  by  introducing  a  single  set  of  states 

to  represent  all  the  transitions  out  of  state  j.  Given  the  transition 

probabilities  for  the  imbedded  chain,  the  (N  -  1)  densities  h  (n)  can 

ij 

be  accumulated  as  in  Eq.  (6.48)  to  obtain  the  y.(n) 

n 

y  .  ( n)  =  p.  .  •  7  h.  .(m)  j  =  1,  2  ...,  N ;  j  /  i  .  (6.53) 

J  ij  1J 

m=o 

The  above  least  squares  procedure  can  be  used  with  these  y^.(n) 

(N  =0,  1,  2,  ...,  T)  as  data.  Varying  numbers  of  auxiliary  states  may 
be  tried  until  a  satisfactory  fit  is  obtained. 

An  example  consisting  of  3  states  is  sketched  in  Figure  6.3. 
States  1,  2,  and  3  comprise  the  original  process;  states  a  and  b  are 
auxiliary  states.  All  possible  transitions  between  states  1,  a,  and  b 
are  allowed,  but  states  2  and  3  cannot  be  left  once  entered.  Twelve 
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FIGURE  6.3  AUXILIARY  STATES  FOR  TRANSITIONS  OUT  OF  1 


transition  probabilities  and  three  row  constraints  are  implied  by  the 
figure  so  that  nine  parameters  are  available  to  approximate  the  holding 
time  behavior  represented  by  the  constants  p p.^,  and  densities 

h12(n)<  h13<n)' 

E.  Discrete  Time  Semi-Markov  Processes 

A  generalization  of  a  Markov  chain  is  a  discrete  time  semi-Markov 
process.  In  addition  to  the  transition  probabilities  required  to  define 
a  Markov  chain,  a  semi-Markov  process  also  requires  the  specification  of 
a  set  of  discrete  conditional  density  functions  h  (n)  to  define  the  time 

ij 

of  the  transition  from  i  to  j,  given  that  the  transition  from  state  i  will 
be  to  state  j.  Thus,  the  transition  out  of  i  can  be  thought  of  as  being 
made  in  two  steps: 

•  The  p^j  determine  the  next  state  (j)  as  for  a  Markov  chain. 

•  The  time  of  the  transition  (measured  from  the  time  state  i 
is  entered)  is  chosen  randomly  from  the  density  function 
hjj  (n) . 

It  is  important  to  note  that  each  semi-Markov  process  also  has  im¬ 
plicit  in  it  a  Markov  chain  called  the  imbedded  Markov  chain.  This 
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imbedded  chain  is  determined  by  the  transition  probabilities  p  alone. 

ij 

That  is,  the  imbedded  chain  concerns  the  sequences  of  states  entered  by 
the  process  and  ignores  times  of  transition.  The  probabilities  of  ab¬ 
sorption  in  Absorbing  states  are  the  same  for  a  semi-Markov  process  and 
its  imbedded  Markov  chain,  but  all  dynamic  quantities  such  as  the  den¬ 
sity  function  of  the  time  of  absorption  will  in  general  be  different. 

Parameter  estimation  for  the  imbedded  Markov  chain  of  a  semi-Markov 
process  can  be  accomplished  with  Markov  chain  methods  because  the  transi¬ 
tion  and  the  dynamic  (time-dependent)  behavior  again  factor  as  they  did 
in  a  Markov  chain.  The  first  step  in  fitting  a  semi-Markov  process  to 
observed  data  is  therefore  to  fit  the  imbedded  chain;  if  this  cannot  be 
fit  satisfactorily,  the  semi-Markov  model  is  inappropriate.  The  next 
step  is  to  fit  the  dynamic  behavior;  indeed,  the  approach  of  the  previous 
subsection  can  be  regarded  as  one  systematic  method  to  accomplish  this. 

Other  methods  may  also  be  used  for  estimating  the  parameters  of  the 

h  (n) .  Theoretical  reasons  may  sometimes  be  found  for  assuming  a  func- 
ij 

tional  form  for  the  h  (n);  data  may  then  be  used  to  estimate  such 

ij 

parameters  as  the  mean  and  variance  of  the  functional  form.  Adding 
auxiliary  states  to  a  Markov  chain  to  approximate  the  semi-Markov  process 
is  equivalent  to  assuming  a  functional  form.  The  advantage  of  the  auxili¬ 
ary  state  method  is  that  the  Markov  chain  formulas  can  continue  to  be 
used  on  the  approximating  chain. 

Various  other  methods  for  estimating  parameters  in  a  semi-Markov 
model  were  investigated  during  the  study.  These  results  are  available 
as  internal  memoranda  and  are  therefore  not  included  in  this  report. 

F .  Testing  for  the  Markov  Property 

Statistical  tests  may  be  devised  for  testing  the  reasonableness  of 
the  Markov  assumption  with  the  classical  chi-squared  goodness  of  fit 
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test.  One  form  of  this  test  results  from  forming  "cells"  into  which 
observations  may  fall  and  comparing  the  observed  number  of  observations 
in  each  cell  with  the  expected  number  of  observations  in  each  cell.7 
The  test  statistic  is 

f  -  nrr 
i _ i 

nn 

i 

where 

k  =  number  of  cells 

f  =  observed  number  of  observations  in  cell  i 
i 

tt  =  probability  that  an  observation  falls  in  cell  i 
i 

n  =  number  of  observations. 

2 

The  test  statistic  x.  is  asymptotically  distributed  as  a  chi-squared 

K*"  J. 

random  variable  with  (k  -  1)  degrees  of  freedom.  When  the  fit  is  good, 

the  deviations  (f.  -  nrr.)  are  small;  when  the  fit  is  poor,  they  are  large 

A  confidence  level  (a)  (such  as  0.95)  is  arbitrarily  set,  and  the  hypothe 

sis  that  the  data  are  consistent  with  a  Markov  chain  model  is  rejected  if 
2 

the  value  of  x  .  from  Eq.  (6.54)  exceeds  the  rejection  limit  found  in 
k- 1 

chi-squared  tables  from  the  confidence  level  a  and  parameter  k. 

A  modified  version  of  this  classical  test  can  be  applied  to  a  Markov 
chain  to  obtain  a  test  statistic 

T  n 

X(n-1)T 

t=l  i=l 

which  is  chi-squared  with  (n  -  1)T  degrees  of  freedom  when  all  y.(t)  are 
nonzero.  Definitions  of  the  quantities  included  are: 


i=l 


(6.54) 
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n  =  the  number  of  states 


p.  =  an  estimate  of  p.  (i,j  =  1,  2}  n) 


y  (t)  =  the  predicted  probability  of  being  in  state  j  at  time  t 

j 

given  by 


n 

y.(t)  =  V  y . (t  -  l)p.  .  (j  =  1,  2,  n) 

j  i  ij 

i=l 


y  (t)  =  observed  proportion  of  times  state  i  is  occupied  at  time 
i 

t  (t  =  1,  2,  T) 


N(t)  =  number  of  observations  at  time  t. 


(For  complete  ASW  exercises  N(t)  h  N,  the  number  of  exercises.) 
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VII  IMPROVAB I LI TY  OF  THE  MEASURES  OF  EFFECTIVENESS 


A.  Introduction 

The  orientation  to  measures  of  effectiveness  taken  in  this  report 
is  that  of  classical  statistics:  the  measures  are  definite,  fixed  num¬ 
bers  exist  in  the  abstract  but  are  unknown.  They  can,  however,  be  esti¬ 
mated  from  data — the  analyst's  task  is  to  devise  methods  for  obtaining 
estimators  that  are  good  in  the  statistical  sense.  A  good  estimator 
should  have  a  small  (preferably  zero)  bias,  and  its  variance  should  be 
small.  In  most  ASW  applications,  the  estimators  may  be  assumed  to  be 
unbiased  (the  mean  or  average  value  of  the  estimator  is  the  true  value) 
so  that  estimators  may  be  judged  by  comparing  their  variances.  For 
practical  purposes  the  smaller  the  variance,  the  better  the  estimator. 

This  section  addresses  the  question  of  improvability  of  the  esti¬ 
mators  of  the  four  principal  measures  of  effectiveness  in  terms  of  variance 
reduction  when  the  model  parameters  must  be  estimated  from  fleet  opera¬ 
tional  data  alone.  It  will  be  shown  that: 

(1)  The  probability  of  success  estimator  given  by  the  ratio 
( successes/trials)  is  essentially  unimprovable. 

(2)  The  mean  time  until  failure,  given  that  failure  occurs, 

is  improvable  by  a  factor  roughly  equal  to  the  probability 
of  failure. 

The  manner  in  which  these  conclusions  were  reached  must  be  discussed  to 
properly  qualify  them.  The  first  conclusion  results  from  a  theorem 
(proved  in  Appendix  E)  essentially  stating  that  the  "cancellation"  noted 
in  conditional  probability  models  (see  Section  III)  also  occurs  in  any 
Markov  model.  The  second  conclusion  results  from  a  second  theorem,  also 
proved  in  Appendix  E. 
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B.  Approximating  an  Arbitrary  Model  by  a  Markov  Model 

In  view  of  the  material  in  the  previous  sections  on  parameter  esti¬ 
mation,  it  seems  reasonable  to  assume  that  a  Markov  chain  model  can  be 
constructed  to  approximate  a  wide  variety  of  other  kinds  of  models.  For 
example,  a  Monte  Carlo  simulation  model  may  be  regarded  as  a  Markov  model 
whose  states  are  defined  in  terms  of  the  model's  variables.  The  descrip¬ 
tion  of  the  state  space  would  be  extremely  difficult  for  large  models 
owing  to  the  very  large  number  of  states.  There  are  so  many  states  that 
a  given  state  has  almost  no  chance  of  being  reentered  on  any  given  repli¬ 
cation.  Nevertheless,  the  basic  structure  is  Markov  and  the  number  of 
states  is  finite. 

The  first  step  in  the  argument,  then,  is  to  assert  that  the  arbi¬ 
trary  effectiveness  model  being  considered  can  be  satisfactorily  approxi¬ 
mated  by  a  Markov  chain  model  with'  a  finite  number  of  states. 

C.  Applying  Theorem  1 

When  transition  data  are  available  to  estimate  the  parameters  of  the 
approximating  Markov  chain,  the  only  estimating  method  needed  is  the 
simple  estimator 


In.  ,/n. 
ij  i 


for  n  >  0 
i 


for  n  =  0 
i 


(7.1) 


where  the  n  are  the  observed  numbers  of  transitions  from  state  i  to 
ij 

state  j  and  the  n  are  the  sums  E  n  In  the  section  where  this  esti- 

1  J  ij 

mator  was  introduced,  it  was  stated  that  the  quantities  p  are  optimal 

ij 

according  to  both  the  maximum  likelihood  and  the  minimum  chi-squared 
criteria;  furthermore,  they  are  unbiased  and  of  minimum  variance  among 
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the  unbiased  estimators.  It  is  therefore  unlikely  that  the  p  can  be 

ij 

improved  in  the  statistical  sense.  Thus,  it  appears  reasonable  that  the 

measures  of  effectiveness  calculated  from  the  approximating  Markov  model 

when  the  true  parameters  p  are  replaced  by  their  estimates  p  [obtained 

ij  ij 

from  fleet  operational  data  by  using  Eq.  (7.1)]  will  be  as  good  as  or 

superior  to  other  estimators  in  the  statistical  sense.  This  is  the 

second  step  in  the  argument.  [It  is  recognized  that  the  second  step 

lacks  rigor:  the  p  are  not  necessarily  maximum  likelihood  estimators 

ij 

for  the  overall  (measures  of  effectiveness)  parameters,  and  maximum  like¬ 
lihood  estimators  are  not  always  minimum  variance  estimators.] 


The  third  step  of  the  argument  leading  to  the  conclusions  (1)  and 
(2)  results  from  the  respective  application  of  the  theorem  and  the  con¬ 
jecture.  The  theorem  states  that  the  absorption  probabilities  obtained 

—  ♦ 

from  the  Markov  chain  calculation  p  =  [(I  -  Q)  R]  when  the  parameters 

s  Is 

of  Q  and  R  are  the  estimators  p  from  Eq.  (7.1)  are  identical  to  the 

ij  - 


ratios 


P 


k 


number  of  times  absorbed  in  state  k 
number  of  trials 


} 


provided  that  the  process  always  starts  in  state  1  and  ends  in  some 
absorbing  state.  In  operational  terms,  all  ASW  exercises  start  in  a 
designated  starting  state  and  continue  until  the  outcome  of  the  exercise  „ 
is  resolved,  that  is,  until  an  event  occurs  to  terminate  the  exercise 
according  to  rules  agreed  upon  ahead  of  time. 

The  second  theorem  is  similar  in  form  to  the  theorem  1  but  concerns 
the  mean  time  to  failure,  given  that  failure  occurs.  However,  a  crucial 


The  subscripts  (l,s  and  l,f)  denote  the  element  in  the  first  row  and 
the  sth,fth  columns  respectively  of  the  indicated  matrix. 
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change  is  needed  in  the  estimation  procedure.  Instead  of  all  the  data 

being  used  to  estimate  the  p .,  only  the  data  for  paths  that  terminate  in 

failure  are  used.  The  conjecture  is:  the  mean  time  until  fail  (given 

* 

that  failure  ultimately  occurs)  calculated  from  the  Markov  formula 

Tf  =  [d  -  Q)_1d  -  Q)_lR]lf/alf 

is  identical  to  the  ratio 

—  sum  of  the  numbers  of  transitions  before  Failure  occurs 

T  =  - 

f  number  of  trials  on  which  Failure  occurred 

D.  Implications  of  the  Theorem  1 

The  implications  of  theorem  1  are  numerous  and  can  be  viewed  in  both 
positive  and  negative  senses.  On  the  negative  side,  the  theorem  implies 
that  to  construct  a  Markov  chain  model  to  obtain  an  improved  estimate  of 
P(success)  is  entirely  futile  when  only  data  obtained  from  complete  (start 
to  finish)  exercises  can  be  used  because  the  result  is  identical  to  the 
simpler  estimate  (number  of  successes/number  of  trials).  At  the  outset 
of  the  current  research,  this  limitation  was  not  recognized;  the  theorem 
came  as  a  surprise  and  somewhat  changed  the  direction  of  the  research 
effort. 

Viewed  positively,  some  implications  of  the  theorem  are: 

•  The  probability  of  success  estimator  can  be  improved  (in  the 
variance  reduction  sense)  if  other  data  are  available  to  im¬ 
prove  the  estimates  of  the  P^j. 

•  The  number  of  states  in  the  Markov  model  and  the  manner  in 
which  they  are  defined  are  entirely  arbitrary  concerning 
the  estimation  of  ps.  State  definitions  may  therefore  be 
made  based  on  the  need  to  obtain  better  estimates  of  the 
other  measures  of  effectiveness. 
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•  The  Markov  property — apparently  required  to  derive  the 
formula  for  pg — is  not  at  all  required.  That  is,  the  esti¬ 
mator  (I  -  Q)“1r  is  very  "robust"  in  that  it  is  completely 
insensitive  to  the  actual  nature  of  the  random  process 
generating  the  sequences  of  states. 

•  It  also  follows  that  postexercise  analysis  of  states  and 
their  transitions  is  not  a  critical  matter  because  state 
definitions  are  arbitrary.  In  particular,  how  the  inevitable 
borderline  cases  are  treated  in  analysis  is  unimportant  so 
long  as  each  trial  traced  a  path  from  the  starting  node  to 
the  correct  terminal  node. 

•  Arguments  sometimes  used  to  show  the  need  for  "aggregating" 
states  to  obtain  sufficient  data  for  estimating  transition 
probabilities  are  invalid  for  the  probability  of  success 
measure — the  number  of  states  and  their  definitions  are  en¬ 
tirely  arbitrary,  and  the  variance  of  the  pg  estimator  is 
independent  of  the  state  space. 


Many  of  the  same  implications  hold  for  the  second  theorem  regarding 
the  conditional  mean  time  until  fail  measure.  Overall,  the  implications 
are  more  favorable  because  the  mean  time  estimator  uses  only  part  of  the 
data  available  (those  paths  leading  to  failure);  the  remainder  of  the 
data  can  be  used  to  improve  the  estimators  p  and  hence  the  conditional 

ij 

A  A 

mean  time  till  failure  T  derived  from  the  p  .  The  extent  to  which  the 

f  ij 

Markov  property  is  required  for  improving  the  p  is  currently  unknown, 

ij 

although  experimental  work  discussed  in  a  later  section  appears  to  indi¬ 
cate  that  this  property  is  not  required.  All  that  is  apparently  required 
is  homogeneity  in  the  sense  that  the  transitions  in  the  transient  portion 
of  the  Markov  chain  model  should  be  generated  by  a  random  process  inde¬ 
pendent  of  where  the  process  is  finally  absorbed. 


The  factor  by  which  the  variance  of  T  can  be  reduced  should  be 


roughly  pf  =  probability  of  fail.  This  follows  because  p^  is  the  approxi¬ 
mate  fraction  of  the  data  that  is  used  to  estimate  T  and  the  variance  of 
—  2  2 

the  sample  mean  T^  is  cr^/N,  where  is  the  true  variance  of  the  time  to 
fail  (T^)  and  N  is  the  number  of  trials.  Experimentation  with  numerical 
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examples 
ties  may 


in  non-Markovian  situations  with  the  required  homogeneity  proper- 
shed  some  light  on  this  variance  reduction  question. 


VIII  PROBABILITY  STATE  TIME  DIAGRAMS 


This  study  largely  concerns  the  development  of  methods  for  applying 
Markov,  semi-Markov,  or  modified  Markov  methods  to  real  operational 
problems.  Because  any  real  operational  problems  are  unlikely  to  be 
exactly  Markovian,  it  is  necessary  to  demonstrate  that  Markov  methods 
can  be  used  in  non-Markovian  situations. 

Three  main  possibilities  to  experiment  with  to  obtain  data  from 
non-Markovian  processes  are: 

•  Operational  data  from  ASW  exercises 

•  Simulated  data  from  digital  computer  simulation  runs 

•  Artificially  constructed  problems. 

Operational  data  of  the  type  needed  for  this  study  are  either  unavailable 
in  a  convenient  form  or  not  available  at  all.  Transition  and  dynamic 
data  (such  as  mean  times  between  states  and  transition  counts  n  )  avail- 

ij 

able  from  sources  such  as  the  UPTIDE  analysis  volumes  are  time-consuming 
to  extract;  therefore,  the  use  of  operational  data  was  ruled  out  for  the 
current  study.  Data  from  simulation  are  easier  to  obtain,  but  their  use 
would  make  analysis  of  Markov  methods  application  more  difficult  because 
the  true  probabilistic  dynamic  structure  of  the  simulation  model  is  in¬ 
variably  unknown.  Thus  the  third  alternative,  the  use  of  artificially 

* 

constructed  dynamic  probabilistic  problems,  was  adopted.  Although  the 
problems  constructed  for  study  had  their  own  special  strongly  non- 
Markovian  structure,  methods  that  did  not  exploit  it  were  always  sought. 


Some  effort  was  also  devoted  to  applying  the  methods  to  data  from  a 
simulation  model  (see  Section  XII). 
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That  is,  the  study  of  artificial  problems  with  special  structure  was 
hopefully  expected  to  find  methods  of  more  general  applicability. 

To  create  artificial  dynamic  probabilistic  problems  sufficiently 
complex  to  test  the  adequacy  of  Markov  approximation  and  sufficiently 
simple  to  make  reasonably  straightforward  calculations  of  the  true  values 
of  the  quantities  of  interest  is  no  simple  matter.  After  some  experi¬ 
mentation,  a  suitable  method  was  found:  a  dynamic  probabilistic  model 
is  defined  by  a  Probability  State  Time  (PST)  diagram.  From  the  PST  dia¬ 
gram  the  many  conditional  probabilities,  probabilities  of  state  as  a 
function  of  time,  and  dynamic  quantities  such  as  the  mean  and  variance 
of  the  transition  time  between  selected  pairs  of  states  can  be  readily 
determined  in  an  easily  understandable  way. 

The  concept  of  defining  a  random  process  by  a  diagram  is  basically 
simple:  assume  that  the  process  depends  upon  a  single  random  variable  u, 

which  is  assumed  to  be  uniform  on  the  interval  (0,1),  This  variable  is 
to  be  associated  with  the  vertical  dimension  on  the  diagram.  The  hori¬ 
zontal  dimension  is  time  (t),  and  the  state  changes  are  specified  merely 
by  drawing  a  line  in  the  (u, t)  plane.  Thus,  a  random  process  including 
the  states  of  opportunity,  detection,  classif cation,  attack,  and  kill  is 
defined  in  Figure  8.1.  A  sample  function  of  the  random  process,  which 
corresponds  to  an  ASW  tactical  encounter  or  exercise,  is  now  determined 
from  the  diagram  by  choosing  a  value  of  u  at  random,  and  drawing  a  hori¬ 
zontal  line  u  units  from  the  top  of  the  rectangle.  Whenever  this  line 
intersects  one  of  the  lines  defining  a  state  change,  a  transition  into 
the  designated  state  occurs  and  the  time  at  which  the  transition  occurs 
is  the  corresponding  value  of  t.  Thus,  line  (T)  represents  the  sample 
function  for  which  detection  occurs  at  time  t  ,  classification  occurs  at 

time  t  ,  attack  occurs  at  time  t  ,  and  failure  occurs  at  time  t  .  Note 
2  3  4 

that  kill  does  not  occur  because  the  K  line  representing  a  transition 
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FIGURE  8.1  ILLUSTRATIVE  PROBABILITY  STATE  TIME  DIAGRAM 


into  the  kill  state  is  not  intersected  by  line  ^1, .  From  t  =  0  to  t  =  t  , 
the  state  is  opportunity;  from  t  to  t  ,  the  state  is  detect;  in  general, 
the  most  recent  state  change  determines  the  state  at  time  t,  which  is  the 
first  intersection  to  the  left  of  the  point  on  the  PST  diagram.  The 
sample  function  represented  by  line  (2)  detects  and  classifies  but  does 
not  attack  and  therefore  fails.  The  sample  function  for  line  detects 

but  does  not  classify  and  therefore  fails.  All  fails  in  this  example 

» 

occur  at  some  fixed  time  large  enough  to  represent  the  maximum  length  of 
an  exercise.  (The  instructions  to  the  hypothetical  exercise  participants 
were  to  continue  trying  to  achieve  mission  success  by  proceeding  through 
the  states  in  sequence  until  this  maximum  time,  with  failure  resulting 
if  kill  has  not  occurred  by  the  maximum  time.) 

The  slanted  horizontal  lines  representing  state  changes  in  Figure 
8.1  represent  conditional  uniform  distributions.  That  is,  the  time  of 
detection  is  uniformly  distributed  given  that  detection  occurs, 
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classification  time  (the  time  classification  occurs)  is  uniformly  dis¬ 
tributed  given  that  classification  occurs,  and  so  forth.  However,  not 
all  these  distributions  are  independent;  on  the  contrary,  they  are  highly 
dependent  because  all  transition  times  are  all  uniquely  determined  when 
u  is  known. 

Conditional  probabilities  are  easily  read  from  the  PST  diagram  by 
taking  ratios  of  lengths  of  appropriate  line  segments.  The  probability 
of  detection  (given  opportunity,  represented  by  the  vertical  line  at  t  =  0) 
is  the  ratio  ab/ac  as  shown  on  Figure  8.2,  while  the  other  conditional 


FIGURE  8.2  RATIOS  OF  LINE  SEGMENTS  DETERMINE  CONDITIONAL 
PROBABILITIES 


122 


probabilities  are 


P(C/D)  =  ef/ab 
P(A/C)  =  gh/ef 
P(K/C)  -  ij/gh 
P(K/0)  =  ij/ac. 

Many  other  conditional  probabilities,  including  those  dependent  on  con¬ 
ditioning  on  knowledge  of  a  state  at  an  earlier  time,  can  be  found 
similarly. 

A  random  process  defined  by  a  PST  diagram  has  a  special  structure 
because  the  probability  densities  for  the  transition  times  are  not  inde¬ 
pendent.  (The  independence  of  these  densities  for  Markov  and  semi-Markov 
processes  is  one  of  the  most  convenient  properties  of  a  Markov  model.) 
Interdependence  is  expected  to  be  one  of  the  major  difficulties  to  be 
faced  in  applications  of  Markov  methods  so  that  experimenting  with  Markov 
methods  on  problems  with  some  form  of  interdependence  was  considered 
appropriate. 

Even  a  Markov  chain  can  be  approximated  by  a  PST  diagram,  however. 
Figure  8.3  is  the  transition  diagram  of  a  six-state  Markov  chain,  with 
transition  probabilities  shown  on  the  branches  representing  the  allowed 
transitions.  In  Figure  8.4,  a  PST  diagram  shows  the  density  function  for 


FIGURE  8.3  MARKOV  PROCESS  WITH  SIX  STATES 
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FIGURE  8.4  PROBABILITY  STATE  TIME  DIAGRAM  FOR  FIRST  STATE  OF  PROCESS 
IN  FIGURE  8.3 

the  transition  from  Opportunity  to  Detect.  (The  line  is  drawn  for  a 
continuous  exponential  distribution  instead  of  a  discrete  geometric  dis¬ 
tribution  for  convenience.  ) 

Since  the  time  from  detect  till  classification  in  the  Markov  chain 
is  geometrically  distributed  given  detection  (i.e.,  the  time  in  the  de¬ 
tect  state  is  geometrically  distributed),  a  curve  shaped  like  that  in 
Figure  8.4  must  be  repeated  once  for  each  value  of  u  so  that  the  density 
of  time  to  classify — given  detection — can  be  independent  of  the  time  to 
detect — given  opportunity.  This  is  impossible  geometrically  so  an  ap¬ 
proximation  must  be  used.  The  detection  curve  can  be  approximated  by  a 
number  of  vertical  line  segments  and  a  classification  curve  can  be  drawn 
for  each  such  vertical  line  segment  as  shown  in  Figure  8.5.  The  resulting 
set  of  lines  can  approximate  the  true  transition  lines  as  accurately  as 


124 


t-*' 


FIGURE  8.5  DETECT  LINE  QUANTIZED;  CLASSIFY  EXPONENTIALS  ADDED 
IN  EACH  SEGMENT 


desired  by  choosing  intervals  sufficiently  small.  Obviously,  this  pro¬ 
cedure  must  be  repeated  again  for  later  states,  with  the  subdivisions 
getting  finer  and  finer  and  the  curves  for  later  transitions  becoming 
increasingly  more  wild.  It  is  not  proposed  here  that  these  approxima¬ 
tions  actually  be  carried  out.  Figure  8.5  is  shown  merely  to  demonstrate 
that  a  Markov  chain  can  be  approximated  by  a  PST  diagram. 

Just  as  each  Markov  process  has  a  Markov  chain  imbedded  within  it, 
a  PST  diagram  has  a  random  process  imbedded  within  it.  In  general,  the 
imbedded  PST  process  is  not  Markovian.  (In  some  cases,  as  when  the  im¬ 
bedded  chain  has  no  loops,  it  will  be  Markovian.  The  first  example 
studied  in  this  report  has  an  imbedded  chain  that  is  Markovian,  but  the 
second  example  does  not.) 
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It  may  be  mentioned  that  in  certain  respects  a  random  process  de¬ 
fined  by  a  PST  diagram  is  more  general  than  a  Markov  chain,  and  in  other 
respects  is  less  general.  The  correspondence  between  the  two  classes  of 
processes  is  best  seen  by  thinking  of  paths  through  the  states  as  the 
basic  elements,  not  the  states  and  the  transition  probabilities. 

A  Markov  chain  containing  loops  has  infinitely  many  paths,  each  with 
probability  equal  to  the  product  of  the  probabilities  along  the  path. 

The  PST-defined  process  has  finitely  many  paths,  and  each  of  these  paths 
has  a  probability  determined  by  the  diagram. 


If  a  PST  diagram  had  infinitely  many  paths,  it  would  be  a  Markov 
chain  if  the  assigned  probabilities  for  each  path  could  be  represented 
as  products  of  probabilities  along  the  respective  paths.  In  general  this 
would  be  impossible  to  do,  implying  that  the  class  of  PST-defined  processes 
with  infinitely  many  paths  is  richer  than  the  class  of  Markov  chains  over 
the  same  set  of  states.  On  the  other  hand,  if  a  Markov  chain  is  limited 
to  a  finite  number  of  paths  on  the  grounds  that  all  finite  problems  neces¬ 
sarily  end  in  a  finite  period  of  time,  then  the  PST-defined  process  is 
somewhat  less  general. 


An  example  will 
process  shown  in  the 


clarify  the  comparison.  Consider  the  two-state 
sketch  below.  The  path  probabilities  for  three 


selected  short  paths  are: 


Probability 
Path  Markov  PST 


11 

(1  - 

P> 

P1 

12 

(1  - 

P)P 

P2 
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P 

P3 
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The  probabilities  of  the  three  paths  may  be  independently  assigned 
for  a  PST  definition;  let  their  values  be  p^,  p^,  p_^  as  shown.  Then  the 
processes  can  be  the  same  only  if  the  probabilities  for  these  paths  agree, 
that  is, 


(1  - 


=  P 


1 


(1  -  p)p  =  p2 


P 


P 


3 


If  these  equations  have  a  solution,  it  must  be  p  =  p,  p  =  ( 1  -  p  )  , 

3  1  3 

and  p  =  (1  -  p  )p  so  that  p  determines  p  and  p  .  Hence  p  ,  p  are 
2  3  3  3  1  2  i*  2 

not  arbitrary  and  the  Markov  chain  is  seen  to  be  more  restricted  than 
the  PST-defined  process. 


There  are  further  computational  advantages  of  the  PST  diagram  method 
for  defining  a  random  process.  Measuring  ratios  of  line  segments  has 
shown  that  certain  conditional  probabilities  are  easily  determined,  and 
other  conditional  probabilities  will  be  found  determinable  by  essentially 
the  same  method.  The  condition  will  usually  define  a  subset  of  the 
probability  space  which  is  composed  of  the  union  of  a  number  of  disjoint 
line  segments,  and  the  event  of  interest  will  be  yet  another  union  of 
disjoint  line  segments.  The  conditional  probability  is  then  the  ratio 
of  the  total  length  of  the  line  segments. 

The  dynamic  structure  of  the  model,  primarily  represented  by  the 
set  of  probabilities  of  occupying  a  state  as  a  function  of  time,  is  easy 
to  determine  from  a  PST  diagram.  To  determine  the  probability  of  being 
in  the  detect  state  at  time  t,  for  example,  requires  summing  the  lengths 
of  all  vertical  line  segments  at  t  which  represent  the  detect  state. 
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Since  the  total  length  (over  all  states)  of  the  line  segments  is  unity, 
the  sum  of  lengths  gives  the  desired  probability  directly. 

Certain  kinds  of  dependence  can  be  incorporated  into  a  problem  de¬ 
fined  by  a  PST  diagram.  When  compensating  factors  that  tend  to  make  the 
sum  of  two  successive  transition  times  a  constant,  for  example,  the  hori¬ 
zontal  distance  between  the  two  state  change  lines  is  that  constant.  An 
operational  example  illustrating  this  sort  of  dependence  might  be  the  de¬ 
pendence  of  time  to  classify,  given  detection,  for  detections  delayed  by 
an  inattentive  sonar  operator.  If  detection  occurs  late  because  of  lack 
of  attention  to  the  sonar  scope,  the  signal  to  noise  ratio  may  be  high 
at  detection,  implying  a  shorter  time  to  classify  than  normal  detections. 
Thus,  the  sum  (time  to  classify,  given  detection)  +  (time  to  detect, 
given  opportunity)  may  tend  to  be  nearer  a  constant  than  an  assumption 
of  independence  of  stages  would  imply. 

Although  it  is  not  being  suggested  here  that  actual  ASW  operational 
problems  are  of  the  type  defined  by  PST  diagrams,  it  is  possible  to  take 
a  (necessarily  finite)  set  of  real  operational  data  (or  simulated  data) 
to  form  the  analogue  of  a  histogram  in  PST  form.  Data  required  are  a 
set  of  paths,  each  consisting  of  a  sequence  of  states  in  the  order  they 
were  entered  and  the  transition  times  for  each  path.  Each  path  could 
then  be  represented  by  a  series  of  dots  on  a  horizontal  line,  as  in  Figure 
8.6.  The  probability  variable  u  has  no  value  to  associate  with  a  path, 
which  poses  a  problem  for  plotting  the  vertical  dimension.  If  all  paths 
are  considered  equally  likely,  the  paths  should  be  separated  by  the  same 
distance,  with  the  ordering  remaining  indeterminate.  In  seeking  structure 
in  the  process,  one  can  experiment  with  various  ways  of  ordering  the 
paths.  (If  structure  can  be  found,  the  interesting  problem  of  interpreting 
the  probability  dimension  could  then  be  undertaken.  Intuition  suggests 
that  the  probability  variable  is  in  some  sense  a  generalized  environmental 
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FIGURE  8.6  HISTOGRAM  FOR  A  PROBABILITY  STATE  TIME  DIAGRAM  FROM 
A  FINITE  SET  OF  DATA 

variable.)  Figure  8.6  shows  a  hypothetical  example  consisting  of  13 
sample  paths  for  a  process  with  four  states.  Dots  represent  data  points 
and  connecting  lines  represent  a  possible  form  for  the  full  problem. 


IX  NUMERICAL  EXPERIMENTATION:  EXAMPLE  ONE 


A.  An  Artificial  Process 

In  this  section,  a  six-state  random  process  is  defined  by  a  PST 
diagram  so  that  various  methods  of  estimation  may  be  tried  and  compared. 
The  structure  is  quite  simple  (no  loops  are  allowed);  each  transition 
must  be  either  to  the  next  state  on  the  path  to  the  kill  state  or  to 
fail.  Figure  9.1  shows  the  PST  diagram.  The  vertical  line  at  the  left 
represents  the  opportunity  state  at  t  =  0.  Transitions  to  intermediate 
states  detect,  classify,  and  attack  are  to  the  right,  and  the  rightmost 
lines  represent  entry  into  one  of  the  two  trapping  states  kill  and  fail. 
Because  of  its  choice  of  states  and  lack  of  loops,  this  example  may  be 
considered  to  represent  the  classical  Conditional  probability  model  of 
ASW.  The  imbedded  chain  is  a  Markov  chain,  but  the  entire  process  is 
strongly  non-Markovian  by  its  very  construction. 

In  this  artificial  example,  no  attempt  was  made  to  choose  realistic 
values  for  the  time  required  to  change  from  state  to  state.  As  can  be 
noted  from  the  form  of  the  diagram,  all  distributions  representing  times 
between  states  are  conditionally  uniform.  For  example,  the  time  to  de¬ 
tect  (give  that  detection  occurs)  is  uniform  from  time  0  to  time  20  units. 
The  time  to  classify,  given  that  classification  occurs,  is  also  uniform 
from  0  to  20  time  units.  A  maximum  time  of  20  units  in  any  nontrapping 
state  is  allowed  and  thus  implies  that  failure  occurs  if  the  next  state 
is  not  reached  in  20  time  units. 

Conditional  probabilities  implied  by  the  diagram  are  readily  found 
to  be  those  in  Table  9.1.  This  set  of  transition  probabilities  and  the 
possible  transitions  are  also  shown  in  Figure  9.2. 
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FIGURE  9.1  PROBABILITY  STATE  TIME 
(NON-MARKOV)  PROCESS  \ 


T 


AGRAM  FOR  AN  ARTIFICIAL  STOCHASTIC 
H  STATES  O,  D,  C,  A,  AND  F 
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Table  9.1 


TRANSITION  PROBABILITIES  DERIVED  FROM  FIGURE  9.1 


P(D/0) 

= 

9/10 

(Detection  occurs 

for  0  S  u  ^ 

0.9) 

P(C/D) 

= 

8/9 

(Classification  occurs  for  0  ^ 

u  ^  0.8) 

P(A/C) 

= 

6/8 

(Attack  occurs  for 

0  <•  u  <,  0.6) 

P(K/C) 

= 

5/6 

(Kill  occurs  for  0 

<,  u  <•  0.5) 

P(F/0) 

= 

1/10 

(0.9  S  u  S  1.0) 

P(F/D) 

= 

1/9 

(0.8  ^  u  ^  0.9  out 

o 

VI 

3 

VI 

o 

o 

.9) 

P(F/C) 

= 

2/8 

(0.6  <.  u  5  0.8  out 

of  0  :£  u  :£  0 

00 

V— ' 

P(F/A) 

= 

1/6 

(0.5  <  u  s  0.6  out 

of  0  ^  u  S  0 

.6) 

FIGURE  9.2  TRANSITION  DIAGRAM  FOR  THE  IMBEDDED  PROCESS  WITH  CONDITIONAL 
PROBABILITIES  FROM  FIGURE  9.1 


Any  exercise  starting  in  opportunity  and  ending  in  kill  or  fail  can 
be  thought  of  as  a  path  through  this  Markov  diagram;  transitions  from 
state  to  state  can  be  tabulated  for  estimating  conditional  probabilities, 
and  these  estimates  can  be  used  as  in  the  WSE  model  to  estimate  the  proba¬ 
bility  of  kill  given  opportunity.  The  probability  of  kill  given  oppor¬ 
tunity  is  the  product  of  the  conditional  probabilities  along  the  single 
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path  from  opportunity  to  kill.  When  the  true  transition  probabilities 
are  used,  the  estimator  for  p^  gives 

9  8  6  5 

P(K/0)  =  —  X  -  X  -  X  -  =  5/10  =  0.5  .  (9.1) 

iu  y  o  o 

Examination  of  the  PST  diagram  in  Figure  9.1  shows  that  Kill  occurs  when 
0  5  u  ^  0.5  and  Failure  occurs  otherwise;  hence,  the  conditional  proba¬ 
bility  estimator  in  Eq.  (9.1)  is  exact  (when  infinite  data  are  available) 
since  the  distribution  of  u  is  assumed  uniform  on  (0,1).  It  appears, 
therefore,  that  a  Markov  model  of  this  process  has  been  successfully 
determined  since  all  conditional  probabilities  and  the  measure  of  effec¬ 
tiveness  P(K/0)  are  correct. 

B.  The  Dynamics  of  the  Process 

Thus  far  the  behavior  of  the  process  in  time  has  not  been  considered. 
From  the  PST  diagram  of  Figure  9.1,  the  probabilities  of  occupying  each 
state  as  a  function  of  time  can  be  determined  by  drawing  a  vertical  line 
at  time  t  and  measuring  the  lengths  of  the  line  segments  representing 
each  state  along  this  vertical  line.  The  resultant  probabilities  of 
state  as  a  function  of  time  are  shown  in  Figure  9.3  for  all  six  states 
for  all  times  from  0  to  60. 

Several  attempts  have  been  made  to  fit  a  dynamic  Markov  model  to 
this  non-Markovian  process  by  a  variety  of  methods.  The  methods  are  to 
estimate  by: 

(1)  Matching  average  time-in-state,  unconditionally. 

(2)  Matching  average  time-in-state,  conditioned  on  the  next 
state . 

(3)  Introducing  additional  states  to  represent  delays. 

(4)  Using  actual  P(state)  as  a  function  of  t  and  least  squares. 
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FIGURE  9.3  COMPARISON  OF  P(STATE)  AS  A  FUNCTION  OF  TIME  CURVES 
FOR  THE  ACTUAL  PROCESS  AND  MARKOV  APPROXIMATION 


(5)  Fitting  a  semi-Markov  process,  after  introducing  addi¬ 
tional  states  as  needed. 

(6)  Defining  a  new  process  whose  states  incorporate  time  in 
the  state  definition. 

These  methods  will  be  discussed  in  turn.  In  all  cases,  infinite  data  are 
assumed  to  avoid  dealing  concurrently  with  sampling  problems  and  struc¬ 
tural  problems. 

C.  Six  Markov  Approximations  to  the  Actual  Process 

1.  Matching  Average  Time  Spent  in  a  State 

In  Section  VI,  the  average  time  spent  in  a  state  i  with  self¬ 
transition  probability  p  was  shown  to  be  the  quantity  1/(1  -  p  ), 

ii  ii 

which  when  inverted  results  in  p  =  1  -  1/T  — where  T  is  the  average 

ii  i  i 

time  spent  in  the  state  before  exit.  Transition  probabilities  for  the 

imbedded  process  are  then  each  multiplied  by  (1  -  p  )  to  normalize  them. 

ii 

Average  times- in-state  (T.)  are  readily  calculated  from  the  PST  diagram, 
and  the  p..  and  normalization  calculations  give  the  transition  matrix 
shown  in  Table  9.2  for  the  approximating  Markov  chain.  With  this  set  of 
parameters,  then,  the  approximating  Markov  chain  model  has  the  following 
properties : 

•  The  conditional  probabilities  are  exactly  the  same  as 
the  actual  process  of  Figure  9.1,  and  the  probabilities 
of  kill  and  fail  are  the  same. 

•  The  average  time  in  each  state  (except  fail  and  kill) 
is  the  same  as  the  average  time  for  the  actual  process. 

To  determine  P( state)  as  a  function  of  time,  pn  can  be  calcu¬ 
lated  by  using  values  from  Table  9.2;  the  results  are  plotted  on  Figure 
9.3.  A  comparison  of  the  true  values  with  the  Markov  chain  values  shows 
that  merely  matching  average  times-in-state  is  insufficient  for  a  good 
dynamic  fit  to  the  actual  non-Markovian  process. 
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General  features  of  the  dynamic  fit,  which  tended  to  persist 
for  other  Markov  attempts  also,  are: 

•  The  actual  random  process  may  have  sharp  cutoffs  at 
points  where  the  probabilities  reach  their  final 
values,  but  the  Markov  approximations  reach  their 
final  values  asymptotically. 

•  The  shapes  of  the  approximating  curves  are  reasonable 
but  lack  the  sharp  corners  of  the  actual  process. 

•  The  approximating  P( state)  as  a  function  of  t  curves 
usually  leads  the  actual  curves  for  small  t,  cross 
over  the  actual  curves,  then  lag  the  actual  curves 
thereafter. 

•  Appreciable  probability  remained  in  the  transient 
states  at  t  =  60  (the  point  at  which  all  probability 
has  been  absorbed  in  the  actual  process). 

2.  Matching  Average  Times  in  a  State,  Conditioned  on  the  Next  State 

The  method  given  above  has  the  disadvantage  that  the  average 
waiting  times  in  a  state  are  computed  by  averaging  over  all  states  to 
which  transitions  are  made.  Thus,  the  average  time  in  the  opportunity 
state  i§ 


0.9  X  (average  waiting  time  in  opportunity | detect  occurs) 


4-  0.1  X  (average  waiting  time  in  opportunity  fail  occurs) 


Since  fail  is  an  absorption  state,  it  is  obvious  that  if  only  the  average 
waiting  time  in  opportunity — given  that  detect  occurs — is  used  as  the 
waiting  time,  the  new  p..  should  better  approximate  the  dynamics. 

Carrying  out  the  new  average  time  calculations  and  repeating 
the  process  used  for  the  first  method  resulted  in  a  fit  that  was  only 
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slightly  better  than  that  of  Method  (1),  however.  [The  improvement  was 
so  slight  that  the  dynamic  response  for  Method  (2)  was  not  plotted.] 

3.  Introducing  Delay  States 

In  Section  VI,  it  was  suggested  that  the  waiting  time  density 

of  an  approximating  Markov  chain  could  better  approximate  the  true  density 

if  delays  were  introduced  to  permit  fitting  both  the  mean  waiting  time 

and  its  variance  instead  of  just  the  mean.  The  methods  of  that  section 

were  used  to  determine  numbers  of  delay  states  D  ;  the  new  transition 

i 

matrix  was  raised  to  successive  powers  to  determine  P(state)  as  a  function 
of  time  once  again.  Figure  9.4  shows  the  results  and  compares  them  with 
the  values  from  the  actual  process.  The  opportunity  curve  is  somewhat 
improved  [over  the  curve  of  Method  (1)],  detection  and  classification  are 
considerably  improved  but  still  lead  the  actual  curves,  attack  is  im¬ 
proved  but  leads  the  actual  curve  and  the  peak  probability  is  far  too 
small,  and  kill  is  somewhat  improved.  The  final  state  (fail)  has  a  some¬ 
what  poorer  fit  than  that  obtained  by  Method  (1). 

4.  Using  Actual  P( State)  as  a  Function  of  Time  and  Least  Squares 
A  fourth  method  of  estimating  the  p  was  a  modified  version 

ij 

of  the  least  squares  model  in  Section  VI-C-5.  The  modification  consisted 

of  using  P(state)  as  a  function  of  time  data  for  t  =  0,  5,  10,  ...,  60 

so  that  the  transition  probabilities  being  estimated  were  the  five-step 
(5) 

probabilities  p  .  The  difference  equations  used  were: 

ij 

y1<n)  =  y1<n  -  Dp^  (9.2) 

y2(n)  =  Vn  '  1)P12  +  y2(n  "  1>P22  (9-3) 
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FIGURE  9.4  COMPARISON  OF  P(STATE)  AS  A  FUNCTION  OF  TIME  CURVES 
FOR  THE  ACTUAL  PROCESS  AND  AUXILIARY  DELAY 
STATES  APPROXIMATION 
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(9.4) 


y3<">  =  y»  -  dp23  +  y3<”  -  »p33 


y  (n)  =  y  (n  -  l)p  +  y  (n  -  l)p 
y4  J3  34  y4  44 


(9.5) 


y  (n)  =  )  y . (n  -  l)p  4-  y  (n  -  1) 
5  /  ^  l  i5  5 

1=1 


(9.6> 


y _(n)  =  y.(n  -  l)p  +  y_(n  -  1) 
6  4  46  6 


(9.7) 


where 


y  (n)  =  P(state  =  i  at  time  5n)  from  the  actual  process, 
l 

To  constrain  the  sum  of  probability  out  of  each  node  to  be  unity,  Eq. 

(9.6)  was  modified  by  replacing  the  p  by 

i5 


i5 


=  1 


-  (Pii  + 


Pi, i+1/ 


for  i  =  1 , 


(9.8) 


This  reduces  the  number  of  variables  to  eight  and  ensures  that 


for  all  states  i. 


The  estimates  of  transition  probabilities  are  shown  in  Table 
9.3.  Parentheses  show  values  obtained  by  normalizing  after  setting  the 
(3,5)  element  to  zero  (this  element  was  negative  in  the  least  squares 
solution).  This  five-step  transition  matrix  was  raised  to  successive 
powers  of  n  to  determine  the  predicted  P(state)  as  a  function  of  time  in 
the  usual  manner;  the  results  are  shown  in  Figure  9.5. 
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FIGURE  9.5  COMPARISON  OF  P(STATE)  AS  A  FUNCTION  OF  TIME  CURVES 

FOR  THE  ACTUAL  PROCESS  AND  LEAST  SQUARES  APPROXIMATION 
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Table  9.3 


(5) 

ESTIMATES  OF  p  USING  P( STATE)  AS  A  FUNCTION  OF  TIME 

ij 


1/j 

1 

2 

3 

4 

5 

6 

1 

0.683 

0.317 

0 

0 

0 

0 

2 

0 

0.6039 

0.2762 

0 

0. 1119 

0 

3 

0 

0 

(0.620) 

(0.380) 

(0) 

0 

4 

0 

0 

0 

0.6186 

0.1421 

0 

5 

0 

0 

0 

0 

1 

0 

6 

0 

0 

0 

0 

0 

1 

Because  the  input  data  leading  to  these  estimates  contained  no 

transition  information,  it  is  of  interest  to  see  what  estimate  of  the 

probability  of  kill  p  is  implied  by  these  p  In  contrast  with  the 

K  ij 

true  value  p  =0.5,  the  estimate  is  readily  found  to  be  p  =  0.432. 

K  K 

Therefore,  it  appears  that  estimators  optimal  for  estimating  the  absorp¬ 
tion  probability  p  may  not  be  optimal  for  predicting  P(state)  as  a  func- 
K 

tion  of  time  when  Markov  methods  are  applied  to  non-Markov  situations. 

5 .  Semi-Markov  Approximation 

The  first  four  methods  used  successively  better  estimates  of 
the  densities  of  waiting  time  in  a  state  without  incorporating  some  of 
the  many  dependent  aspects  of  the  true  model  into  the  estimators.  Methods 
(5)  and  (6)  used  auxiliary  states  defined  so  as  to  remove  some  of  the  de¬ 
pendencies.  An  additional  refinement  in  Method  (5)  was  that  the  waiting 
times  were  not  required  to  be  geometrically  distributed.  That  is,  the 
fifth  approximation  was  by  a  semi-Markov  process,  not  by  a  Markov  chain. 

The  fifth  attempt  to  "Markovize"  the  actual  process  used  a 
semi-Markov  model  with  28  states — the  original  6  and  22  others.  Density 
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functions  of  the  transitions  were  allowed  to  vary  according  to  the  state 
being  entered  (semi-Markov  property).  The  Z  transform  method  was  used 
for  the  calculations  and  was  carried  out  by  hand.  Because  of  the  labor 
entailed,  only  a  few  comparisons  were  made.  These  are  given  in  Table  9.4. 


Table  9.4 

RESULTS  FROM  SEMI-MARKOV  APPROXIMATION — 28  STATES 


Time 

State 

P( state) [true] 

P( state) [ semi-Markov] 

Error 

5 

Detect 

0.2 

0.  273 

— 

10 

De tect 

0.38 

0.4 

■  1 

20 

Detect 

0.53 

0.537 

v  ■ '  • ; 

30 

De tect 

0.145 

0.145 

0 

35 

Detect 

0.045 

0.045 

0 

40 

Detect 

° 

0 

0 

16 

Classify 

0.19 

0.  222 

0.03 

20 

Classify 

0.29 

0.3164 

0.04 

40 

Classify 

0.23 

0.2182 

0.01 

30 

Attack 

0.  255 

0.  224 

0.03 

40 

Attack 

0.42 

0.36 

0.06 

50 

Attack 

0.26 

0.33 

0.07 

The  dynamic  behavior  of  the  semi-Markov  model  approximates  the 
actual  behavior  considerably  better  than  any  of  the  earlier  Markov  chain 
approximations  used  in  Methods  (1)  through  (4).  It  is  encouraging  to  see 
that  a  semi-Markov  model  can  fit  such  a  complex  dynamic  probabilistic 
process  as  that  defined  by  the  PST  diagram.  However,  the  semi-Markov 
model  used  in  Method  (5)  used  knowledge  of  the  actual  process  itself; 
this  knowledge  will  not  be  available  in  applications  to  real  problems. 
(All  other  approximating  methods  in  this  report  use  only  observed  data 
so  that  this  fitting  attempt  was  an  exception  to  the  general  rule:  use 
the  artificial  problems  to  find  promising  methods  that  can  be  transferred 
to  real  problems.) 
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6. 


Incorporating  Time  into  the  State  Definition 


The  estimating  methods  given  so  far  can  all  be  viewed  as  syste¬ 
matic  ways  to  add  auxiliary  states  and  transitions  to  improve  the  fit  of 
the  dynamic  behavior  represented  by  P(state)  as  a  function  of  time.  In 
the  sixth  estimating  method  tried  on  this  problem,  time  itself  was  used 
to  define  auxiliary  states.  The  method  can  be  shown  to  be  equivalent  to 
a  model  with  time-varying  transition  probabilities. 

In  the  original  process,  there  are  six  states  and  time  varied 
from  0  to  60.  In  the  new  process,  the  states  are  the  ordered  pair  (S,T), 
where  "s"  denotes  the  state  in  the  original  process  and  T  is  an  integer 
value  of  time.  For  example,  the  new  state  (C,  12)  or  C12  represents  the 
state  "the  original  process  is  the  classify  state  at  time  12."  The  de¬ 
rived  process  has  6  X  60  =  360  states  because  times  from  1  to  60  are  used. 

Any  sample  path  from  the  original  process,  together  with  the 
transition  times,  now  generates  a  path  in  the  new  process.  For  example, 
the  value  of  u  =  0.15  in  the  original  process  results  in 

Detection  at  time  17 

Classification  at  time  22 

Attack  at  time  31 


Kill 

at  time  52. 

In  the  derived 

process,  the  path  is 

01,  02, 

. .  .  ,  016, 

D17 ,  D18,  . . ., 

D21 ,  C22,  C23, 

...,  C30 ,  A31,  A32, 

•  •  •  y  A5  1  y 

K52,  K53, 

. . . ,  K60. 

A  computer  program  was  written  to  select  a  set  of  sample  paths 
from  the  PST  diagram  and  to  perform  the  processing  necessary  to  estimate 


* 

The  0  =  Opportunity. 
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P( state)  as  a  function  of  time.  When  infinite;  data  are  assumed  to  be 

available,  this  model  fits  the  original  process  essentially  perfectly 

* 

in  the  P( state)  as  a  function  of  time  sense.  This  was  first  hypothe¬ 
sized  and  then  verified  numerically.  (When  u  was  varied  systematically 
from  zero  to  1  in  steps  of  0.001 — i.e.,  one  thousand  paths — the  probability 
of  state  as  a  function  of  time  curves  were  indistinguishable  from  those 
in  Figure  9.3.  A  random  sample  of  1000  paths  also  produced  an  excellent 
fit,  as  expected.  A  random  sample  of  100  paths  produced  estimates  typi¬ 
cally  in  error  by  0.01  to  0.02,  with  occasional  values  in  error  by  as 
much  as  0.05. 

Although  the  derived  process  has  many  more  states  than  the 
original  process  (360  versus  6),  in  many  respects  the  length  of  the  cal¬ 
culations  themselves  do  not  increase  in  proportion.  For  the  six-state 
process,  the  products  of  powers  of  the  6X6  transition  matrix  are  multi¬ 
plied  to  obtain  probabilities  of  state  as  a  function  of  time.  In  the 
derived  process,  matrix  multiplication  was  unnecessary  because  the  proba¬ 
bilities  at  a  state  in  this  process  are  themselves  the  probabilities  of 
being  in  a  state  of  the  original  process  at  the  corresponding  time;  hence, 
the  probabilities  in  this  process  can  be  calculated  by  simple  recursion 
methods  because  of  the  absence  of  loops  in  the  transition  diagram. 

D.  Summary 

A  six-state  dynamic  probabilistic  non-Markov  process  was  defined  by 
a  PST  diagram  in  order  to  have  a  known  process  for  experimentation.  Six 
different  methods  were  used  to  determine  the  parameters  of  a  Markov  or 
semi-Markov  model  to  approximate  the  actual  process.  Of  special  interest 
was  the  dynamic  behavior  of  the  approximating  processes  as  represented  by 


* 

Quantization  is  required  which  introduces  some  small  errors. 
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the  curves  showing  P(state)  as  a  function  of  time  for  each  of  the  six 
states  and  all  times  from  0  to  60. 

The  findings  are  the  following: 

•  P( state)  as  a  function  of  time  fit  the  actual  process  rather 
poorly  for  Markov  chain  Methods  (1)  through  (4)  and  fit  quite 
well  for  semi-Markov  and  nonstationary  Methods  (5)  and  (6). 

•  The  best  fits  were  obtained  by  using  auxiliary  states. 

(Markov  chain  methods  used  auxiliary  states  to  better  repre¬ 
sent  waiting  times  in  a  state,  and  the  semi-Markov  model 
and  nonstationary  model  used  auxiliary  states  to  reduce 
dependency. ) 

•  P( state)  as  a  function  of  time  curves  from  the  approximating 
Markov  chains  were  quite  smooth  and  approached  their  final 
values  asymptotically;  the  sharp  corners  and  abrupt  cutoffs 
of  the  actual  process  were  not  well-approximated. 

•  The  least  squares  method,  which  used  the  actual  P( state)  as 
a  function  of  time  as  inputs,  gave  parameters  that  implied 

a  value  of  probability  of  kill  (p^)  and  differed  appreciably 
from  the  true  value  of  pK.  This  suggests  that  estimators 
which  are  best  for  estimating  pK  may  not  be  best  for  estimating 
P( state)  as  a  function  of  time. 
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X  MARKOV  MODELS  THAT  INCLUDE  FALSE  CONTACTS 


A .  Introduction 

Traditional  conditional  probability  models  of  ASW  system  effective¬ 
ness  do  not  explicitly  include  false  contact  considerations  such  as  the 
detection  and  prosecution  of  false  contacts.  One  of  the  objectives  of 
the  current  study  is  to  develop  models  that  include  false  contacts  in  the 
dynamic  ASW  environment. 

Three  operational  false  contact  models  that  generalize  the  traditional 

conditional  probability  models  are  defined  in  this  section.  Two  different 

approaches  to  defining  a  state  space  are  used;  in  both  cases,  a  single  de- 

* 

tection  opportunity  of  an  ASW  unit  on  an  enemy  submarine  is  assumed. 
Conventional  ASW  screening  by  destroyers  and  use  of  destroyers  in  an 
antisubmarine  barrier  are  examples  of  tactical  situations  to  which  the 
operational  models  apply. 

The  first  model  is  a  Markov  chain  model  that  generalizes  the  tradi¬ 
tional  conditional  model  in  the  simplest  possible  way.  The  second  model 
is  a  modification  of  the  first,  but  it  uses  a  continuous  time,  competing 
process  formulation  for  transitions  into  and  out  of  the  prosecute  false 
contact  state.  Tables  are  given  to  show  how  false  contacts  degrade  ASW 
effectiveness  for  the  second  model.  The  third  model  uses  a  state  space 
whose  states  are  ordered  pairs  of  the  original  states.  The  first  element 
of  the  pair  describes  the  ASW  units  situation  with  respect  to  false  con¬ 
tacts,  and  the  second  element  describes  the  situation  with  respect  to  an 
actual  submarine. 

* 

The  generalization  to  multiple-opportunity  situations  appears  to  be 
straightforward. 
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B.  A  Markov  Chain  Model  for  an  ASW  Unit  in  the  Presence  of 

False  Contacts 

The  diagram  in  Figure  10-1  shows  the  states  for  a  Markov  model  of  a 
single  ASW  unit  when  false  contacts  may  be  present.  It  is  the  simplest, 
meaningful  generalization  of  the  traditional  conditional  probability 
model  with  the  five  states  designated.  It  should  be  noted  that  the  dia¬ 
gram  has  a  single  loop.  Two  states  have  been  added  to  the  traditional 
model;  they  are  succinctly  designated  as  prosecute  false  contact  and 
opportunity  with  false  contact. 

Fuller  definitions  of  these  additional  states  are: 

•  Detect  and  Prosecute  False  Contact:  Detect  a  contact  that 
will  later  be  found  to  be  false  and  take  appropriate  action 
to  prosecute  the  contact. 

•  Opportunity  with  False  Contact:  An  actual  submarine  arrives 
and  presents  a  Detection  Opportunity  while  a  False  Contact 
is  being  prosecuted. 

Therefore  the  prosecute  false  contact  state  is  terminated  in  two  ways: 

•  When  the  contact  has  been  classified  as  a  false  contact  and 
the  ASW  unit  has  returned  to  the  screen. 

•  When  an  actual  submarine  arrives  that  either  presents  a  de¬ 
tection  opportunity  or  would  have  presented  a  detection 
opportunity  to  the  unit  if  it  were  not  prosecuting  a  false 
contact. 

The  model  applies  to  situations  in  which  there  is  a  single  opportunity 
to  detect  a  submarine.  For  example,  a  destroyer  in  the  traditional 
screening  role  with  the  mission  of  detecting,  classifying,  attacking,  and 
killing  a  penetrating  enemy  submarine  fits  this  model.  A  second  example 
is  a  barrier  operation  involving  a  surface  ship  patrolling  one  portion  of 
the  barrier.  An  ASW  unit  in  either  of  these  contexts  may  be  more  than  a 
single  ship;  for  example,  it  may  be  a  destroyer  and  a  LAMPS  helicopter. 
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FIGURE  10.1  TRANSITION  DIAGRAM  FOR  FALSE  CONTACT  MODELS  1  AND  2 


State  transitions  will  be  explained  next.  It  is  assumed  that  the 
starting  state  is  search  (state  1).  While  searching,  false  contacts  may 
from  time  to  time  be  detected  and  prosecuted,  causing  the  state  to  change 
to  state  3  (prosecute  false  contact).  The  transition  from  state  1  to 
state  3  will  depend  on  the  definition  of  false  contact  and  the  ship  de¬ 
tecting  may  or  may  not  make  an  action  appropriate  to  an  actual  submarine. 

Of  the  two  opportunity  states,  state  2  (opportunity)  is  defined  in 
the  usual  way  on  the  basis  of  a  maximum  detection  range  for  the  ASW  unit. 
State  4,  opportunity  with  false  contact  (defined  above),  is  the  second 
opportunity  state.  A  transition  from  state  3  to  state  4  occurs  when  a 
penetrating  submarine  comes  within  the  maximum  detection  range  while  the 
ship  is  prosecuting  a  false  contact.  ♦ 

State  3  also  changes  to  state  4  when  a  submarine  would  have  presented 
a  detection  opportunity  to  the  ASW  unit  had  it  not  changed  position  to 
prosecute  a  false  contact.  It  will  be  necessary  to  be  somewhat  arbitrary 
in  order  to  make  this  definition  more  precise.  If  a  destroyer  has  an 
assigned  area  to  search,  the  submarine  may  be  said  to  present  an  oppor¬ 
tunity  with  false  contact  when  it  arrives  into  this  assigned  area,  for 
example.  If  the  submarine  is  not  detected  while  in  an  opportunity  state, 
then  the  ASW  unit  fails  to  detect  and  hence  enters  the  fail  state.  It 
is  assumed  that  once  opportunity  passes  there  will  not  be  another  chance. 
If,  however,  detection  of  the  submarine  occurs  (the  assumption  here  is 
that  detection  means  detects  and  prosecutes),  then  the  state  changes  to 
detect  just  as  though  the  false  contact  had  not  been  made.  From  detection 
onward,  the  transitions  are  the  same  as  those  in  the  traditional  model. 
Failure  can  occur  at  any  time  following  detection,  and  classification  and 
attack  are  necessary  conditions  for  kill.  The  possibility  of  counter¬ 
attack  is  not  considered  in  this  model. 
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For  the  present,  it  is  assumed  that  data  are  available  to  allow  the 


estimation  of  transition  probabilities  from  a  set  of  tactical  ASW  exer¬ 
cises  or  incidents.  Let  n  be  the  observed  number  of  transitions  from 

ij 

state  i  to  state  j.  Then  the  transition  probabilities  not  involved  with 
the  loop  are  estimated  in  the  traditional  way: 


n 

.  ,  ,  number  of  classifications  56 

p  =  P(C  D)  =  -  =  7 - r 

56  number  of  detections  In  +  n  | 

\  56  58 / 


n 

.  |  number  of  attacks  67 

p  =  P(A  C)  =  -  =  - r 

67  1  number  of  classifications  In  +  n  1 

\  67  68/ 


P79  =  P(K|A) 


n 

number  of  kills  79 


number  of  attacks  in  +  n  ) 

[  79  78/ 


Each  of  these  estimators  is  an  instance  of  the  use  of  the  general  formula 

9 

P  .  =  n  ./  /  n.,  =  n.  ,/n. 

1J  lj/  Immd  lk  1J  1 

k=l 

When  care  is  taken  to  count  transitions  properly,  this  formula  holds  for 

all  transitions  including  those  involved  with  the  loop.  The  denominator 

n  represents  the  total  number  of  transitions  out  of  a  state.  For  the 
i 

transient  states  (i.e.,  all  states  except  kill  and  fail),  n  also  equals 

7  i 

the  number  of  transitions  into  the  state.  Thus,  the  number  of  transitions 
into  or  out  of  search  is 


(number  of  exercises  beginning  in  search)  +  (number  of  times  false 


contacts  are  prosecuted  to  completion)  =  n  +  n 

LI  X  Ol 
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where  n 


is  the  number  of  exercises  starting  in  Search.  Since  a  given 


01 

trial  may  result  in  0,  1,  2,  false  contact  prosecutions,  the  total 

number  of  entries  into  Search  may  have  any  nonnegative  value.  Similarly, 
the  total  number  of  entries  into  state  3  (detect  and  prosecute  false  con¬ 
tact)  will  be  the  denominator  of  the  two  transition  probabilities  corre¬ 
sponding  to  the  two  ways  of  leaving  this  state. 

Given  the  estimates  of  p  =  P[S(n  +  1)  =  j|S(n)  =  i]  for  all  (i.j) 

i  j 

branches  on  the  state  diagram,  Mason's  rule  of  Appendix  D  for  calculating 
transmissions  in  linear  systems  can  be  applied  to  give  the  formula  for  the 
desired  estimate  of  the  probability  of  kill.  [Matrix  inversion  to  obtain 
the  absorption  matrix  A  =  (I  -  Q)  is  equivalent  to  this.]  When  transi- 
tion  data  are  derived  from  a  set  of  complete  exercises,  the  theorem  in 
Appendix  E  ensures  that  this  estimate  is  identical  to  the  simpler  estimate 
(number  of  kills/number  of  exercises). 

Mason's  rule  requires  the  calculation  of  loop  transmissions  and  open 
path  transmissions.  Since  there  is  a  single  loop  with  transmission 
p  p  ,  the  graph  determinant  is  simply 

Iw  O  1 


A 


1  -  p 


D 

13*31 


There  are  two  open  paths  from  search  to  kill.  The  first  path  is 
Search  Prosecute  False  Contact  —  Opportunity  with  False  Contact  -»  De¬ 
tect  -*  Classify  ->  Attack  -*  Kill  with  transmission  equal  to  the  product 

of  the  conditional  probabilities  along  the  path  p  „p  p,  p  p  p  .  The 

F  s  13  34  45  56  67  79 

second  path.  Search  ->  Opportunity  -*  Detect  -•  Classify  ->  Attack  ->  Kill, 

has  transmission  p  p  p  p  p  These  two  open  paths  have  the  detect, 

12  25  56  67  79  * 

classify,  attack,  and  kill  states  in  common  so  the  last  three  transition 

probabilities  can  be  factored  out.  The  formula  for  p  which  results  from 

K 
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Mason's  rule  is  therefore 


P  P 
1  13  34 


U  +  (; 


^P12P25/J 

1  —  p  p 

P13P31 


P  •  P 
56  *67 


(10.1) 


Because  of  the  simplicity  of  the  loop  structure,  this  formula  can 
either  be  interpreted  more  directly,  or,  alternatively,  be  derived  from 
basic  probability  principles.  Examination  of  the  transition  diagram 
shows  that  the  numerator  is  the  probability  of  transmitting  directly  from 
the  search  state  to  the  kill  state,  without  returning  to  the  search  state. 
The  loop  product  i*1  the  denominator  is  the  probability  that  a 

given  trial  starting  in  search  will  result  in  one  or  more  loops  (i.e., 
returns  to  search).  Therefore,  the  graph  determinant  A>  with  value 
1  -  p  p  ,  is  the  probability  of  no  return  to  search  for  a  given  trial. 
Conditioning  on  the  number  of  loops  (k)  before  a  series  of  transitions 
finally  leads  to  one  of  the  absorbing  states. 


P(Kill)  =  /  P(Kill  path  begins  with  k  loops) 


X  P(path  begins  with  k  loops) 


(10.1a) 


The  first  term  in  the  production  on  the  right,  which  is  actually  inde¬ 
pendent  of  k,  is  the  numerator  in  Eq.  (10.1).  The  second  term  in  the 
product  is  simply  (p  p  )  .  Therefore  Eq.  (10.1a)  becomes 

1«J  ol 


P(Kill)  =  [numerator  of  Eq.  (10.1)]  X 


CO 

£(■ 


P13P31/ 


=  numerator/^1  -  p^p. 


(10.1b) 


which  is  identical  to  Eq.  (10.1). 
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An  algebraically  equivalent  form  separates  the  probability  estimate 
Eq.  (10.1)  into  two  components. 


pppppp  ppppp 

13  34  45  56  67  79  12  25  56  67  79 

p  =  - - - - - - -  +  - - - - - x - 

k  1  “  p  •  p  1  “  p  p 

P13  P31  *13*31 


(10.2) 


In  this  form  the  first  term  depends  only  on  false  contacts  and  the  second 

is  the  usual  estimate  divided  by  1  -  p  p  ,  which  may  be  considered  to 

13  31’ 

be  a  false  contact  "correction  term. " 

The  basic  expression  of  Eq.  (10.1)  can  be  rewritten  again  to  show 
another  possible  interpretation.  When  the  search  to  kill  transition  is 
considered  in  two  phases — the  first  from  search  to  detect  and  the  second 
from  detect  to  kill — the  formula  for  the  probability  of  kill  becomes 


P,  =  P(Kill) 
k 


=  P(Detect)  X  P(Ki 11 | Detect) 


P13P34P45  P12P25  \  ... 

1-pp  +  1  -  p  p  j  P56P67P79 

13*31  *13*31/ 


.  (10.3) 


The  effects  of  false  contacts  are  isolated  in  the  first  term  representing 

P(detect),  except  for  the  "correction  factor”  1-pp  in  the  second 
’  13  31 


Several  special  cases  of  Eq,  (10.1)  are  of  interest  and  are  discussed 
as  follows: 

•  When  there  are  no  false  contacts  the  estimate  of  p^3  is  zero 
and  the  kill  probability  formula  is  then  reduced  to  the 
traditional  estimator 


p  =  p  p  p  p  p 

k  12  25  56  67  79 


(10.4) 


When  a  false  contact  prosecution  completely  precludes  de¬ 
tection  of  an  actual  submarine,  p^^  will  be  zero;  then  the 
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formula  becomes 


Pk  = 


P  P  D  D  D 

12*  25  56  67  79 

^  a 

1  -  p  p 

13  31 


(10.5) 


Even  though  the  first  term  is  now  zero,  the  (1  -  p  p  ) 
correction  term  remains  in  the  denominator  to  modify  the 
traditional  estimate. 

•  If  p31  is  zero — that  is,  if  the  ASW  unit  never  returns  to 
search  following  prosecution  of  a  false  contact — the  de¬ 
nominator  disappears  and  pk  is  the  sum  of  two  terms: 


Pk  P13P34P45P56P67P79  +  P12P25P56P67P79 


(10.6) 


Alternatively,  since  p3^  =  1  and  PggPgyPyg  is  common  to 
both  terms, 


A  A  A  .  A  A  AAA 

Pk  "  P13P45  +  P12P25  P56P67P79 


(10.6a) 


The  first  term  (in  parentheses)  shows  that  there  are  two 
exclusive  ways  to  detect;  the  remaining  product  is  the 
probability  of  kill  given  detection. 

•  If  an  actual  submarine  always  arrives  while  the  ASW  unity 
is  in  Search  (and  never  arrives  while  prosecuting  a  false 
contact)  the  p^  and  are  both  zero.  This  implies  that 

P34  =  1  and  p-^  =  1  -  p^g.  Putting  these  into  Eq.  (10.1) 
results  in 


f>  = 


P25P56P67P79 


(10.7) 


and  shows  that  no  false  contact  considerations  remain. 
[The  pl2  factor  disappeared  because  it  was  assumed  that 
the  submarine  always  arrived  during  search  so  that  the 
opportunity  state  (2)  was  always  reached.) 

•  When  false  contacts  are  so  frequent  that  they  are  effec¬ 
tively  always  present,  then  p12  may  be  considered  to  be 
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zero  and  the  formula  becomes 


Pk  = 


A  A  A  A  A  A 

D  D  D  D  D  D 

13  34  45  56  67  79 

_  A  /V 

1  ‘  P13P31 


(10.8) 


If  p^g  is  zero  also,  then  =  1  and  Eq.  (10.8)  becomes 


Pk  = 


p  p  p  p  p 
*34*  45  56  67  79 


1  -  p 


(10.9) 
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C.  A  Mixed  Markov  Model  for  an  ASVV  Unit  in  the  Presence  of  False  Contacts 


The  Markov  chain  model  in  the  previous  section  can  be  generalized 
somewhat  by  passing  to  the  competing  process  formulation  of  a  continuous 
time  Markov  model.  This  model,  which  will  have  transition  rates  as  its 
only  parameters,  can  be  reduced  either  to  the  Markov  chain  model  involving 
only  transition  probabilities  or  to  a  mixed  model  involving  both  transi¬ 
tion  rates  and  transition  probabilities  as  parameters.  The  mixed  model 
should  be  suitable  for  assessing  the  impact  of  false  contacts  on  ASW  sys¬ 
tems  effectiveness  from  data  obtained  from  exercises  in  which  false  con¬ 
tacts  were  not  considered. 


In  general,  the  competing  process  formulation  of  a  continuous  time 
Markov  model  is  as  follows.  Associated  with  each  branch  of  the  transition 
diagram  (i.e.,  with  each  transition)  is  an  exponential  distribution  with 
parameter  \  .  The  set  of  \  completely  characterizes  the  process,  for 

ij  ij 

transitions  are  determined  from  the  \  by  the  following  rules.  First 

ij 

selecting  a  set  of  candidate  transition  times  |T_|,  one  value  for  each 

branch  out  of  state  i.  [The  candidates  are  chosen  independently  of  one 

th 

another,  the  (i,j)  time  being  selected  at  random  from  an  exponential 
distribution  with  parameter  \  .]  The  next  state,  j  ,  is  selected  as 

ij 

that  state  j  with  the  minimum  transition  time,  and  the  time  of  the  tran¬ 
sition  is  chosen  equal  to  that  time.  Symbolically,  j*  =  j  such  that  T 
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is  minimized;  therefore,  T  is  the  waiting  time  in  i  prior  to  the 

lj 

transition  to  j  .  It  can  be  shown  that  the  resulting  distribution  of 
the  time  required  to  transit  from  i  to  j  is  exponential  with  parameter 
X.  ,  independent  of  the  destination  state  j,  where 


(10.10) 


The  transition  probabilities  for  the  imbedded  Markov  chain  implied  by 
the  above  rules  can  be  shown  to  be  simply  related  to  the  transition  rates 

by 


P.  .  =  X.  ./X.  when  i  /  j  .  (10.11) 

ij  U  i 


To  apply  the  competing  process  concept  to  the  formulation  to  a  false 
contact  model,  consider  a  model  with  the  same  set  of  states  as  the  Markov 
chain  model  above.  The  rules  for  writing  down  the  Laplace  transform  of 
the  transmission  from  state  1  (search)  to  state  9  (kill)  result  in  Eq. 
(10.12) . 


Vs) 


*12^25  \aX34\5 

56  67  79 

iivs)(vs) +  (x+s)(vs)(vs)J 

L(vs)(vs)(vs)J 

1  - 


\3X31 

(vs)(vs) 


(10. 12) 


The  derivation  of  this  formula  follows  from  the  use  of  Mason's  rule  with 

an  appropriate  set  of  branch  transmissions.  The  state  1-state  2  branch, 

which  formerly  had  a  transmission  p  ,  now  *las  tbe  transmission 

X  /(\  +  s).  In  general,  the  i  -  j  transition  has  the  transmission 

12  1 

X.  ./(X.  +  s) .  The  Markov  chain  probability  of  kill  formula  is  now  ob- 


ij  i 

tained  by  setting  the  Laplace  variable  s  to  zero  and  substituting  p.^ 


for  X  /X  throughout  the  right  side  of  Eq.  (10.12). 

ij  i 
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The  objective  in  this  section  is  to  obtain  a  modified  model,  how¬ 
ever.  To  this  end,  the  variable  s  is  set  to  zero  and  p  is  substituted 

ij 

for  \  /\  for  the  5-6,  6-7,  and  7-9  transitions  only.  The  result  is 

ij  i 

Eq.  (10.13). 


^12^25  \3S4\5\ 

-  +  - I  p  p  p 

I  56*67  79 


V0>  - 


12 


13  4 


1  - 


\3X31 


(10.13) 


Since  the  arrival  rate  of  an  actual  submarine  should  not  depend  on 

whether  the  ASW  unit  is  prosecuting  a  false  contact  or  is  in  Search,  the 

quantities  \  and  \  may  be  assumed  to  be  equal.  Replacing  \  by  A. 

12  34  34  12 

and  multiplying  numerator  and  denominator  by  \  results  in  Eq.  (10.14). 


V0’ 


/Ss  Ns  Ns' 

M  h  N  N ; 


P56P67P79 


\  - 


X13X31 


(10.14) 


To  have  a  more  convenient  set  of  parameters  to  work  with,  it  is  now 
assumed  that  the  probability  of  detection  is  degraded  by  a  constant  factor 
f  when  a  false  contact  is  being  prosecuted.  Specifically,  it  is  assumed 
that 


p  „  =  p  X  f 

45  25 


(10.15) 


where  f  is  a  fixed  number  in  the  interval  (0,1).  Substitution  of  Eq. 
(10.15)  together  with  Eq.  (10.11)  into  Eq.  (10.14)  gives  Eq.  (10.16). 


W0) 


Naf1  -  f\3A3)p25 
\  ”  ^13Sl/?V3 


P56P67P79 


(10.16) 
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From  the  factorization  in  Eq.  (10. 16),  False  Contacts  can  be  seen  to 

result  in  a  degradation  in  the  transition  probability  p  in  the  tradi- 

15 

tional  conditional  probability  model.  Denoting  this  degraded  value  by 

p  ,  we  have 
15’ 


\3X31 


(10.17) 


The  value  of  the  degradation  factor  itself  is  the  ratio 


d 


(10.18) 


Another  parameter  needs  to  be  introduced  for  the  sake  of  interpret- 
ability .  If  it  assumed  that  the  average  time  spent  prosecuting  a  false 
contact  in  the  absence  of  a  submarine  arrival  (l/\  )  is  a  constant  k 

■  O  X 

times  the  average  time  between  arrivals  of  false  contacts  in  the  absence 
of  a  submarine  arrival  ( l/\  ) ,  then 


k 


) 


or  k  =  \  /\ 


31 


(10.19) 


To  illustrate  the  numerical  value  of  the  degradation  caused  by  false 
contacts,  let  it  first  be  assumed  that  the  value  of  f  is  zero.  This 
means  that  no  submarine  detection  is  possible  while  the  ASW  unit  is  prose¬ 
cuting  a  false  contact.  Without  loss  of  generality  it  may  be  assumed 
that  the  arrival  rate  of  actual  submarines  in  the  absence  of  false  con¬ 
tacts  (\  )  is  unity.  Equation  (10.18)  then  becomes 

1 2 


161 


d  = 


i1  +  \s)  +  1  +  k/K 


(10.20) 


13 


when  \  \  =  1,  f  =  p  /p  =0,  and  k  =  k  /\ 

12  34  '  *45  *25  ’  13  31 

Under  these  conditions,  the  degradation  factor  d  is  given  in  Table 

10.1  for  the  parameters  indicated  when  \  is  also  zero. 
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Table  10.1 

VALUES  OF  DEGRADATION  FACTOR  CAUSED  BY  FALSE  CONTACTS 


Parameters 

Degradation  Factor 

k 

^13 

d 

0.  2 

CD 

0.833 

0.2 

10 

0.836 

0.2 

5 

0.839 

0.  2 

1 

0.857 

0.4 

T> 

0.714 

0.4 

10 

0.722 

0.4 

5 

0.729 

0.4 

1 

0.778 

0.6 

00 

0.625 

0.6 

10 

0.639 

0.6 

5 

0.651 

0.6 

1 

0.727 

0.8 

CD 

0.556 

0.8 

10 

0.574 

0.8 

0.592 

0.8 

0.692 

(\2 

V  = 

K34 

0) 

=  1,  f  =  0, 
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If  we  return  to  the  general  case  where  f  >  0  (arriving  submarines 
may  be  detected  when  a  false  contact  is  being  prosecuted),  the  formula 
may  be  readily  derived 


^  +  f  (1/10  +  (x12ai3)_ 

(\2  +  v)  +  <l/k>  +  (K12/^13) 


(10.21) 


Once  again  \  may  be  chosen  equal  to  unity  to  normalize  the  parameters, 
1 2 

resulting  in  the  final  expression 


1+f  (1/k)  +1(1/X13) 

d  =  - ^ - L -  .  (10.22) 

Some  special  cases  of  Eq.  (10.22)  may  be  noted  as  a  check.  In  all  check 
cases  let  \ — the  failure  rate  when  in  search — be  zero.  When  f  =  1, 
the  value  of  d  becomes  unity  as  it  should  since  false  contacts  do  not  in¬ 
fluence  the  process.  Alternatively,  when  the  value  of  k  is  zero,  the 
prosecution  of  false  contacts  is  instantaneous  and  results  in  a  degrada¬ 
tion  factor  of  unity.  A  third  case,  analogous  to  a  queueing  system  in 
which  the  arrival  rate  equals  the  departure  rate,  results  when  the  value 
of  k  is  unity.  For  the  third  case,  algebraic  manipulation  of  Eq.  (10.22) 
results  in 


1  i  A.. 


i- 


18  (1/k)  +  1/K 


d  = 


(10.23) 


Therefore,  as  the  arrival  rate  of  false  contacts  \  increases,  the 
degradation  factor  approaches  the  value  d  =  (1  +  f)/2. 
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In  Eq.  (10.22),  the  parameters  are  seen  to  be  f,  \  ,  and  the  quan- 

1 J 

tity  (1/k)  +  (l/\  ).  Table  10.2  was  constructed  for  two  values  of  f  and 

.L 

values  of  the  other  parameters  as  indicated. 


Table  10.2 

DEGRADATION  FACTOR  d  CAUSED  BY  THE  PRESENCE  OF  FALSE  CONTACTS 


Degradation 

Parameters 

Factor  d 

1  +  X18 

1 

Hs  + 

1 

k 

f  =  0.5 

f  =  1.0 

1.001 

i 

0.7496 

0.9995 

3 

0.8743 

0.99925 

6 

0.9278 

0.9991 

12 

0.9607 

0.99908 

24 

0.9791 

0.99904 

1.01 

1 

0.7463 

0.99502 

3 

0.8685 

0.99256 

6 

0.9207 

0.9915 

12 

0.9527 

0.99085 

24 

0.9707 

0,99049 

1.5 

1 

0.6 

0.8000 

3 

0.6364 

0.7273 

6 

0.65 

0.7000 

12 

0.6579 

0.6842 

24 

0.6622 

0.6494 

D.  A  False  Contact  Model  Formed  from  a  "Product” 

The  third  false  contact  model  uses  a  different  method,  which  may  be 
called  the  product  method,  for  defining  a  set  of  states.  Again  it  is 
assumed  that  at  most  one  detection  opportunity  is  presented  on  an  actual 
submarine.  The  first  four  of  the  original  states — Opportunity,  Detect, 
Classify,  and  Attack — may  be  considered  to  describe  the  ASW  unit's 
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situation  with  respect  to  either  an  actual  submarine  or  a  false  contact. 

If  subscripts  s  and  f  are  used  to  denote  submarine  and  false  contact 
respectively,  then  the  cartesian  product  of  these  sets  may  be  defined  by 
choosing  one  state  from  each  set  and  adding  the  subscript  f  to  the  first 
component  and  subscript  s  to  the  second  component.  For  convenience  of 
notation,  the  original  states  may  be  denoted  by  their  first  letters. 

For  example,  the  product  state  (0^,0  )  means  there  is  a  detection  oppor¬ 
tunity  on  a  false  contact  and  also  a  detection  opportunity  on  an  actual 
submarine. 

Another  state  needs  to  be  added  to  complete  the  original  actual  sub¬ 
marine  set.  This  state,  designated  N  ,  represents  no  opportunity  for  de¬ 
tection  of  an  actual  submarine.  If  an  ASW  unit  can  handle  one  actual 
submarine  contact  and  one  false  contact  simultaneously,  then  all  combina¬ 
tions  in  the  product  set  of  states  are  possible  although  some  may  rarely 
be  observed  in  actual  ASW  operations.  It  is  simpler,  however,  to  con¬ 
tinue  considering  all  combinations  of  states  rather  than  to  try  eliminating 
those  that  may  be  rarely  observed.  Therefore,  the  set  of  states  is  formed 
by  choosing  the  first  component  from  the  set  [0  ,  C^,  A^,  K^}  and  the 

second  component  from  the  set  [N  .  0  .  D  ,  C  ,  A  ],  and  adding  two  more 

s  s  s  s  s 

states  fail  (not  to  kill  the  actual  submarine)  and  kill  (the  actual  sub¬ 
marine).  There  are  27  states  in  all,  and  the  starting  state  may  be  taken 

to  be  (0  N  )  when  the  submarine  is  initially  beyond  the  maximum  detection 
f  s 

range  of  the  ASW  unit. 

It  should  be  noted  that  not  only  the  detection  of  false  contacts  but 
also  the  misclassif ication  of  false  contacts  and  attacks  on  false  con¬ 
tacts  are  possible  in  this  model.  A  complete  sequence  of  errors  of  the 
latter  kind  are  represented  by  the  sequence  of  states 

(v0>  (vNs)’  (Vs)’  (vNs)’  (vNs)  • 
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The  last  state  in  this  sequence  represents  the  kill  of  the  false  contact, 
which  may  have  been  evaluated  as  kill  of  an  actual  submarine  during  the 
operation.  The  other  kind  of  error,  calling  an  actual  submarine  a  false 
contact,  is  also  possible.  These  errors  are  represented  in  the  model  by 
transitions  to  the  fail  state  from  any  state  with  second  component  D 

s 

(detect  actual  submarine). 

Examples  of  this  application  to  some  of  the  other  states  will  now 
be  discussed.  If  the  exercise  is  begun  with  a  destroyer  in  a  barrier 
search  before  the  actual  penetration  attempt  by  a  submarine,  then  the 
starting  state  can  be  taken  as  (0^,N  )  since  there  are  always  opportuni¬ 
ties  to  detect  false  contacts.  The  state  (0  ,0  )  would  then  be  entered 

f’  s 

when  the  submarine  comes  within  detection  range  of  the  destroyer's  sen¬ 
sor  (s)  because  Opportunities  are  then  presented  for  detecting  either  the 
submarine  or  a  false  contact.  Detecting  a  false  contact  first  would 

cause  a  transition  to  (D  ,0  );  and  if  the  submarine  moves  out  of  detection 

f  ’  s 

range  before  the  destroyer  classifies  the  false  contact,  the  state  becomes 
(Df,N  ).  If  another  detection  opportunity  never  materializes  on  the  sub, 
failure  will  result.  (The  exact  point  at  which  the  state  should  become  F 
is  somewhat  arbitrary.) 

In  general,  when  an  actual  submarine  is  being  prosecuted,  the 
Opportunity  -*  Detect  -*  Classify  -*  Attack  -*  Kill  chain  can  be  followed 
with  the  first  coordinate  always  being  0^  as  the  submarine  is  progressively 
detected,  classified,  attacked,  and  killed.  If,  for  example,  a  false  con¬ 
tact  is  detected  while  an  attack  is  under  way  (weapon  in  the  water)  and 
this  causes  the  destroyer  to  direct  its  attention  to  the  false  contact 
(which  may  be  a  decoy),  then  the  state  becomes  (D  ,Ag).  The  sub-f 
sequence  for  the  false  contact  may  follow,  but  it  will  be  interrupted 
if  kill  of  the  submarine  actually  occurs.  (The  state  then  becomes  K  to 
end  the  exercise.)  If  the  torpedo  expires  without  killing,  then  the 
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second  component  of  state  becomes  N  if  the  submarine  has  passed  out  of 

s 

detection  range  or  becomes  0  if  it  remains  within  range. 

s 

Some  examples  may  be  helpful  at  this  point.  Consider  three  hypo¬ 
thetical,  reconstructed  exercises  that  evolved  as  follows. 


The  destroyer  detects  a  false  contact,  classifies  it  as  non¬ 
sub,  and  returns  to  search.  The  submarine  comes  within  de¬ 
tection  range  but  is  undetected;  thus,  the  mission  results 
in  a  failure.  The  state  sequence  is:  (0  ,N  ),  (D  ,N  ), 

f '  c  '  f  ’  c  > 

<VV'  (VV'  F- 

The  destroyer  detects  a  false  contact  after  searching  for 
some  time;  while  the  false  contact  is  being  classified,  the 
actual  submarine  comes  within  range.  The  false  contact  is 
correctly  classified  as  false  and  the  destroyer  returns  to 
search  while  the  submarine  is  still  within  range.  The  sub¬ 
marine  passes  out  of  range  without  being  detected.  The 
state  sequence  is: 


<VV’  F- 


‘W’  <VV’  <VV’  "VV’ 


•  The  destroyer  detects  a  false  contact,  correctly  classifies 
it,  and  returns  to  search.  It  then  detects  another  false 
contact,  classifies  it  as  a  submarine,  and  performs  an 
attack  on  the  false  contact.  The  destroyer  then  returns 
to  search,  the  actual  submarine  comes  within  range  and  is 
detected.  The  submarine  is  classified  as  a  submarine  and 
attack  maneuvering  begins.  The  submarine  evades  success¬ 
fully  before  weapon  launch,  and  no  further  detection  oppor¬ 
tunity  is  available  against  it.  The  state  sequence  is: 
(Of,Ns>,  (Dt,Ns),  (Cf,Ks>,  (0f,»s>,  <Df,B5),  (Cf,Ns), 

<W«  <0f’V>  ‘W’  (VDs>’  ‘W-  F- 


Markov  models  earlier  in  this  report  were  sufficiently  small  to  per¬ 
mit  the  calculation  of  absorption  probabilities  by  inspecting  the  transi¬ 
tion  diagram  and  applying  Mason's  rule.  There  are  far  too  many  states  in 
this  model  for  this  approach  to  succeed.  Instead,  the  algebraic  formula 
for  the  absorption  matrix  A  derived  in  Section  V  must  be  used: 


A 


(I  - 


Q)_1R 


(10.24) 
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where  the  states  are  systematically  numbered  such  that  the  transition 


probability  matrix  P  can  be  partitioned  in  the  form 


(10.25) 


In  this  model,  Q  is  dimensioned  25  by  25,  R  is  25  by  2,  and  the  identity 
matrix  is  2  by  2.  Calculations  for  the  other  measures  of  effectiveness 
may  also  be  carried  out  in  matrix  terms  as  shown  in  Section  V. 

It  is  assumed  in  Eq.  (10.24)  that  transition  probability  estimates 
are  available  from  data.  If  the  n  data  are  available,  the  formula 

ij 

p  =  n  /n  may  be  used  to  obtain  the  elements  of  Q  and  R.  Other  data 
ij  ij  i 

such  as  average  time- in-state  data  may  also  be  used  to  estimate  the  p 

ij 

as  discussed  in  Section  VI. 
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XI  NUMERICAL  EXPERIMENTATION:  EXAMPLE  TWO  (FALSE  CONTACTS) 


A.  Introduction 

This  section  is  concerned  with  numerical  experimentation  on  the 
Markov  chain  false  contact  model  defined  in  Section  X.  A  PST  diagram 
is  used  to  define  a  non-Markovian  dynamic  probabilistic  process  with  the 
same  nine  states  as  used  for  the  Markov  chain  model.  Data  of  various 
kinds  are  generated  from  the  PST  diagram  and  are  used  to  estimate  parame¬ 
ters  of  the  Markov  model.  These  parameters  are  then  used  to  calculate 
the  estimates  [P(state)  as  a  function  of  time]  from  the  Markov  model,  and 
comparison  of  these  curves  with  those  from  the  actual  (PST-def ined) 
process  reveals  the  adequacy  of  the  Markov  and  semi-Markov  approximations. 

Because  the  actual  process  is  so  strongly  non-Markovian,  the  state 
space  had  to  be  extended  in  an  attempt  to  "decorrelate"  some  of  the  de¬ 
pendencies  in  the  structure.  Auxiliary  states  were  systematically  added 
by  three  different  methods.  The  first  method  added  states  to  provide 
"memory"  of  the  previous  state.  The  second  method  removed  the  loop  by 
adding  a  new  state  each  time  a  state  was  reentered.  The  third  method 
added  states  to  provide  memory  of  all  previous  states. 

A  summary  of  the  rather  lengthy  experimentation  on  this  process  will 
now  be  given.  It  was  shown  in  Section  VII  that  the  Markov  chain  calcula¬ 
tion  for  the  absorption  probability  (p  )  is  exact  when  the  true  transi- 

K 

tion  probabilities  are  known  for  the  imbedded  process  even  when  the  true 
process  is  non-Markovian.  Knowing  these  true  probabilities  is  equivalent, 
in  terms  of  sample  size,  to  having  an  infinite  number  of  samples  of  the 
process.  In  the  first  set  of  experiments  in  Section  XI-B,  finite  samples 
were  first  chosen  from  the  true  process,  and  all  transition  probabilities 
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were  estimated  from  the  finite  sample  by  using  the  estimator 


A 

P. 

1J 


finally,  the  p  formula  was  used  to  estimate  the  true  probability  of  kill, 
k 

The  estimator  was  verified  to  be  identical  in  all  cases  to  the  simple 
direct  estimator  (number  of  kills)/(number  of  samples).  Next,  some  simple 
statistical  tests  were  made  on  the  transition  probabilities  and  the  esti¬ 
mator  p  and  all  results  agreed  with  predictions  from  binomial  theory, 
k 


The  next  set  of  experiments  (Section  XI-C)  concerns  methods  for 
choosing  the  parameters  of  a  Markov  or  semi-Markov  model  by  using  the 
given  set  of  states  in  order  to  fit  the  dynamic  behavior  [P(state)  as  a 
function  of  time]  to  the  observed  data  from  the  process  defined  by  the 
PST  diagram.  Because  the  theorem  on  absorption  probabilities  already 
has  proved  that  no  improvement  is  possible  in  estimating  the  probability 
of  absorption  in  state  9  (kill)  for  a  set  of  complete  exercises,  the  de¬ 
conditioning  can  help  only  when  the  P( state)  as  a  function  of  time  fit  is 
of  interest  or  when  not  all  data  are  from  a  set  of  complete  exercises. 

In  the  latter  case,  better  estimates  of  probability  of  kill  as  well  as 
dynamic  behavior  should  be  possible  because  of  the  deconditioning.  The 
following  methods  were  used. 


•  Markov  chain  with  original  states,  fitting  mean  waiting  time 

T  in  each  state, 
i 

•  Semi-Markov  model,  using  geometrically  distributed  T.  . . 


•  Semi-Markov  model, 


using  uniformly 


distributed  T 

ij 


•  Semi-Markov  model,  with  transition  time  density  hij(*) 
tained  by  averaging  across  the  true  process. 


ob- 


•  Markov  chain,  with  transition  probabilities  selected  so  that 
P(state)  as  a  function  of  time  fits  the  data  in  a  least 
squares  sense. 
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For  comparison  purposes,  samples  of  size  100  were  also  drawn  from  the 
true  population  and  the  P(state)  at  time  t  estimated  by  the  observed 
proportion  of  times  the  process  was  in  that  state  at  time  t. 

The  next  experiments  for  fitting  P( state)  as  a  function  of  t  used 
altered  sets  of  states.  Additional  states  were  introduced  systematically 
in  three  ways.  In  the  first  method  (Section  XI-D-1)  states  are  redefined 
to  correspond  to  branches  of  the  original  process,  which  is  equivalent 
to  a  model  with  a  one-state  memory.  The  second  method  (Section  XI-D-2) 
removes  the  loop  in  the  model  by  considering  a  return  to  the  search  state 
as  a  new  state  on  each  return.  The  third  method  (Section  XI-D-3)  is 
again  semi-Markov  and  provides  complete  memory  of  the  earlier  states. 

The  effect  in  each  case  was  to  add  more  states  by  "conditioning,"  thereby 
increasing  the  number  of  transition  probabilities  to  be  estimated  which 
in  theory  should  allow  one  to  find  a  better  fit  in  the  P(state)  as  a  func¬ 
tion  of  time  sense.  These  three  methods  produced  a  somewhat  better  fit 
than  those  using  the  original  set  of  states. 

In  Section  XI-E,  the  Markov  property  was  tested,  A  comparison  of 
several  statistical  quantities  from  the  approximating  Markov  model  and 
the  true  non-Markovian  model  were  first  given.  Estimates  of  the  proba¬ 
bility  of  occupying  any  state  not  in  a  loop — including  the  success  state — 
are  found  to  be  exact  for  any  finite  set  of  complete  exercise  data  while 
estimates  of  most  other  quantities  differ  in  the  true  process  and  the 
approximating  Markov  process.  The  theorem  of  Section  VII  proved  in 
Appendix  E  shows  equality  of  the  measures  because  the  signals  at  the  nodes 
were  closely  related  to  the  probability  of  entering  a  state  in  the  formu¬ 
lation  for  the  theorem.  A  chi-squared  test  for  determining  whether  the 
entire  set  of  data  may  be  assumed  to  come  from  a  Markov  model  concludes 
the  section. 
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PROBABILITY 


Figure  11.1  shows  the  PST  diagram  for  the  nine-state  false  contact 
model  of  Section  X.  Not  all  possible  kinds  of  paths  are  represented  by 


the  PST  diagram  (the  1-2-5-6-7-8  path  is  missing,  for  example).  A  maxi¬ 
mum  of  three  trips  around  the  single  loop  is  implied  by  the  diagram;  they 
occur  for  0.4  ^  u  ^  0.5.  The  probability  of  kill  is  0.19  because  the  kill 
state  is  entered  in  three  places  (0.1  £  u  ^  0.2,  0.40  ^  u  ^  0.42, 


2 


0.70  £  u  ^  0.77)  and  the  lengths  of  these  intervals  adds  up  to  0.19. 

Time  varies  from  t  =  0  to  t  =  200  in  an  unspecified  set  of  units. 

Several  quantities  of  interest  from  the  actual  process  are  given  in 
the  following  tables  and  figures.  Figure  11.2  shows  the  transition  dia¬ 
gram  for  the  imbedded  process  and  its  transition  probabilities  that  would 
be  obtained  from  sampling  the  process  infinitely  often.  These  transition 
probabilities  are  also  shown  in  tabular  form  in  Table  11.1.  Also  shown 
on  the  branches  are  the  numbers  of  transitions  (based  on  1000  equally 
spaced  paths)  that  determine  the  true  transition  probabilities  for  the 
imbedded  chain.  Table  11.2  shows  the  true  probabilities  of  occupying 
all  states  as  a  function  of  time;  the  several  Markov  approximations  to  the 
actual  process  will  be  judged  by  determining  how  well  the  approximating 
P( state)  as  a  function  of  time  curves  match  these  data.  Mean  times  of 
the  i  to  j  transition  (T  )  data  are  shown  in  Table  11.3  for  all  states 

ij 

i  and  j;  the  standard  deviations  of  the  T  are  shown  in  Table  11.4. 

ij 

B.  Monte  Carlo  Experiments  for  Estimation  of  Transition  Probabilities 

For  this  set  of  experiments,  the  imbedded  process  is  sampled;  that 

is,  transition  counts  n.  .  are  obtained  by  sampling,  but  the  information 

on  transition  times  between  states  is  ignored.  The  natural  estimator 

p  =  n  /£  n  is  used  to  estimate  the  transition  probabilities  p 
ij  ij  k  ij  ij 

Ten  sets  of  replications  of  sizes  25  to  100  were  independently  run  and 
the  p  estimates  are  tabulated  in  Tables  11.5  and  11.6.  A  single  column 

ij 

of  each  of  these  tables  represents  estimates  for  a  set  of  replications 
(or  case).  The  estimates  of  the  measure  of  effectiveness  (probability  of 
terminating  in  the  kill  state)  resulting  from  Eq.  (10.1)  of  Section  X-B 
are  shown  on  the  bottom  row. 

The  rightmost  columns  compare  the  average  p  .'s,  where  the  averaging 
was  over  all  ten  cases,  with  the  true  transition  probabilities.  The 
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OF  FALSE  CONTACT  MODEL 


Table  11.1 


TRANSITION  PROBABILITY  MATRIX  (p  )  FOR  IMBEDDED  CHAIN 

ij 


1 

2 

3 

4 

5 

6 

7 

8 

9 

1 

0 

0.288 

0.548 

0 

0 

0 

0 

0.164 

0 

2 

0 

0 

0 

0 

0.529 

0 

0 

0.471 

0 

3 

0.794 

0 

0 

0.206 

0 

0 

0 

0 

0 

4 

0 

0 

0 

0 

0.850 

0 

0 

0.150 

0 

5 

0 

0 

0 

0 

0 

0.795 

0 

0.205 

0 

6 

0 

0 

0 

0 

0 

0 

0.686 

0.314 

0 

7 

0 

0 

0 

0 

0 

0 

0 

0.208 

0.792 

8 

0 

0 

0 

0 

0 

0 

0 

1.000 

0 

9 

0 

0 

0 

0 

0 

0 

0 

0 

1.000 

estimated  p  obtained  by  using  the  averaged  p  is  0. 1806  for  25  replica- 
k  ij 

tions/case  set  of  data  and  0.194256  for  the  100  replications/case  set  of 

data.  The  true  value  of  p  is  0.19. 

k 

The  last  row  of  each  table  shows  the  number  of  successes  for  each 

case;  the  ratio  (number  of  successes) /(number  of  trials)  may  be  directly 

compared  with  the  value  of  p  computed  for  the  formula  for  absorption 

K 

probabilities.  The  correspondence  is  exact  for  the  25  replications  cases 
while  some  round-off  error  occurs  in  the  fourth  decimal  place  for  the  100 
replications  cases. 


A  simple  test  on  the  reasonableness  of  the  estimates  of  p  can  be 

K 

made  on  the  basis  of  the  observation  that  the  variance  of  a  binomial  dis¬ 
tribution  with  n  trials  and  probability  of  success  p  is  p  (1  -  p  )/n. 

K  K  K 

In  the  25-sample  case,  the  true  variance  is  0.19  (1  -  0.19)/25  =  0.006156 
so  that  the  standard  deviation  is  0.0783.  If  a  normal  approximation  to 
the  binomial  density  is  used,  about  68  percent  of  the  samples  within  one 
standard  deviation  of  the  true  mean  value  0.19  can  be  expected;  that  is, 
in  the  interval  (0.1117,  0.2683).  Of  the  ten  estimates,  two  are  outside 
and  eight  are  inside  this  interval,  which  is  quite  reasonable  since 
(0.68)  x  10  s  7  were  expected.  Repeating  this  procedure  for  the  100-sample 
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Table  11.2 


PROBABILITY  OF  BEING  IN  A  STATE  AS  A  FUNCTION  OF  TIME  x  1000 


State 


t 

1 

2 

3 

4 

5 

6 

7 

8 

9 

5 

950 

0 

50 

0 

0 

0 

0 

0 

0 

Id 

90  0 

0 

1  0  0 

0 

0 

0 

0 

0 

0 

1  5 

850 

0 

1  50 

0 

0 

0 

0 

0 

0 

20 

80  0 

0 

20  0 

0 

0 

0 

0 

0 

0 

25 

780 

0 

220 

0 

0 

0 

0 

0 

0 

30 

760 

0 

2  HO 

0 

0 

0 

0 

0 

0 

35 

7H0 

0 

260 

0 

0 

o 

0 

0 

0 

HO 

720 

0 

280 

0 

0 

0 

0 

0 

0 

H5 

70  0 

0 

30  0 

0 

0 

0 

0 

0 

0 

50 

680 

0 

320 

0 

0 

0 

0 

0 

0 

55 

6  6  0 

0 

2H0 

1  0  0 

0 

0 

0 

o 

0 

60 

6H0 

0 

1  60 

172 

•'“i  o 

C.  1 

0 

0 

0 

0 

65 

6H5 

u.  J 

130 

137 

58 

5 

0 

0 

0 

70 

650 

50 

1  0  0 

1  0  1 

6H 

!  *7 

o 

0 

0 

75 

6  5  0 

75 

-7tr 

b  b 

70 

35 

29 

0 

0 

80 

650 

1  0  0 

50 

30 

“?  “*? 

4  3 

50 

0 

0 

85 

538 

1  87 

75 

30 

H8 

52 

70 

0 

0 

90 

H50 

250 

1  0  0 

30 

30 

H9 

91 

0 

0 

95 

H0  0 

268 

.1  25 

30 

7'7 

HO 

H8 

0 

52 

1  0  0 

350 

262 

150 

30 

68 

HO 

30 

0 

70 

105 

350 

256 

125 

30 

99 

HO 

30 

0 

70 

1  10 

350 

250 

1  0  0 

30 

1  30 

HO 

30 

0 

70 

1  1  5 

32H 

2H3 

1  0  1 

30 

1  62 

HO 

30 

0 

70 

1  20 

280 

237 

1  20 

30 

1  93 

HO 

30 

0 

70 

1  25 

2H3 

20  6 

1  57 

30 

1  6  1 

1  0  3 

30 

0 

70 

130 

230 

2  0  0 

170 

30 

1  0  5 

1  65 

30 

0 

70 

1  35 

230 

20  0 

1  70 

30 

80 

1  52 

68 

0 

70 

1  HO 

230 

20  0 

1  70 

30 

80 

90 

1  30 

0 

70 

1H5 

"7  "7  Cj 

212 

CT  *7 

•  J 

30 

80 

90 

1  0  5 

o 

95 

1  50 

340 

225 

35 

30 

so 

90 

8  0 

0 

1  20 

1  55 

315 

267 

1  8 

30 

80 

90 

55 

0 

1  H5 

1  60 

290 

3 1  0 

0 

30 

so 

90 

30 

0 

1  70 

165 

290 

310 

0 

30 

so 

90 

30 

0 

1  70 

1  70 

290 

295 

0 

30 

95 

90 

30 

0 

1  70 

175 

290 

2Hu 

0 

30 

150 

90 

30 

0 

1  70 

1  80 

290 

2H0 

0 

30 

1  50 

90 

30 

[1 

1  70 

1  85 

290 

2  HO 

0 

30 

95 

1  H5 

30 

0 

1  70 

1  90 

290 

2H0 

0 

30 

90 

1  1  0 

70 

0 

1  70 

195 

290 

2H0 

0 

30 

90 

1  10 

55 

o 

1  85 

20  0 

0 

0 

0 

0 

0 

0 

0 

8 1  0 

1  90 
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MEAN  TRANSITION  TIMES  (T 
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ESTIMATES  FOR  p.  .—25  REPLICATIONS  FOR  EACH  CASE 


178 


Note:  Formula  for  estimating  kill  probability: 


Table  11.6 


179 


cases  gives  an  interval  (0.1508,  0.2292)  with  estimates  for  six  cases 
lying  inside  the  interval  and  estimates  for  four  cases  lying  outside  the 
interval,  which  is  again  satisfactory. 

Similar  distribution  tests  were  also  made  on  the  distribution  of  the 
transition  probability  estimates  p  .  The  variance  is  more  difficult  to 

ij 

determine  this  time,  however,  since  the  number  of  samples  is  random.  If 
N  is  the  random  number  of  samples  available  to  estimate  p 

ij 

var  p  =  E  j~var  ^p  |  N jTj  +  var Je^p  |  =  E^var^p  .  |>ljj 

because  E(p  | N)  =  p  ,  a  constant.  With  the  use  of  an  average  value 
ij  1  ij 

assumption,  the  variance  of  p  ,  is  approximately 

var  p  =  p  (l  -  p  )/N  , 

ij  ij\  ij/ 

where  N  is  the  average  value  of  N. 

Table  11.7  summarizes  the  results  of  the  distribution  calculations 
for  the  100-repl ication  cases.  The  true  values  of  p  are  shown  in  the 

ij 

left  column,  the  next  column  gives  the  sample  standard  deviations  (ft  ) 

ij 

of  p  ,  while  the  last  column  shows  the  numbers  of  cases  (out  of  ten) 

ij 

whose  estimates  fell  inside  the  estimated  one-sigma  band  about  p  .  The 

ij 

average  number  lying  inside  was  7.2,  compared  with  the  theoretical  ex¬ 
pected  value  of  6.8.  It  is  of  interest  to  note  that  the  variance  of  p 

k 

is  0.0392,  which  is  smaller  than  the  estimated  sigmas — nine  of  the  twelve 
p  from  which  it  is  computed. 

ij 

C.  Approximating  the  Actual  Process  by  Markov  and  Semi-Markov  Processes 

Several  attempts  were  made  to  approximate  the  dynamic  performance  of 
the  actual  (PST-def ined)  process  by  selecting  parameters  for  a  Markov  or 
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Table  11.7 

NUMBERS  OF  SAMPLES  WITHIN  THE  ONE-SIGMA  INTERVAL 


Transition 

(i,  j) 

True  p 

ij 

Standard  Deviation 

& 

ij 

Number  Samples  Inside 

(p.  P.  .  +  &.  .) 

ij  ij  ij  ij 

12 

0.  288 

0.0340 

7 

13 

0.548 

0.0340 

7 

18 

0.  164 

0.0340 

6 

25 

0.529 

0.0699 

7 

28 

0.471 

0.0699 

7 

31 

'  0.794 

0.041 

8 

34 

0.206 

0.041 

8 

45 

0.850 

0.0798 

7 

48 

0.150 

0.0798 

8 

56 

0.795 

0.0609 

6 

58 

0.205 

0.0609 

2 

67 

0.686 

0.0784 

6 

68 

0.314 

0.0784 

7 

78 

0.  208 

0.0828 

9 

79 

0.792 

0.0828 

9 

average  =7.2 

Note:  Expected  number  of  samples  inside  the  interval  is  6.8  based 

on  normal  approximation. 

The  variance  of  p.  is  0.0392. 

k 


semi-Markov  process  by  methods  discussed  in  Section  VI.  Unless  otherwise 
indicated,  an  unlimited  amount  of  data  were  assumed  to  be  available  from 
the  actual  process. 
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Since  the  P(state)  curves  characterize  the  dynamics  of  the  process, 
the  approximating  Markov  process  should  have  similar  P( state)  as  a  func¬ 
tion  of  t  curves.  The  true  probability  of  state  as  a  function  of  time 
relationships,  which  were  shown  in  tabular  form  in  Table  11.2,  are  shown 
in  graphical  form  in  Figure  11.3.  The  Markov  approximations  are  expected 
to  have  difficulty  achieving  a  fit  to  these  rather  irregular  curves. 

1 .  Markov  Chain  Model  with  Matching  Average  Waiting  Time 

In  this  first  attempt  to  match  the  dynamic  behavior  of  the 

actual  (PST-def ined)  process,  the  simple  method  of  estimating  the  p  for 

a  Markov  chain  used  the  actual  values  of  the  transition  probabilities  for 

the  imbedded  chain  together  with  the  average  waiting  time  T  in  each  state 

i 

i.  Mean  times  were  found  by  a  weighted  average  across  rows  in  Table  11.3. 
The  method,  which  gave  a  poor  fit  in  the  earlier  numerical  example  in 
Section  IX,  gave  an  even  poorer  fit  in  this  example.  The  approximating 
P(state)  as  a  function  of  t  curves  were  so  poor  that  they  were  not  plotted. 
Evidently,  better  methods  are  needed. 

2.  The  First  Semi-Markov  Approximation:  Geometrically 
Distributed  T  — 

The  second  attempt  at  fitting  P( state)  as  a  function  of  time 

employed  a  semi-Markov  model  with  geometrically  distributed  transition 

times  T  .  The  parameter  was  chosen  to  match  the  known  mean  transition 
_ij 

times  T  from  Table  11.3.  As  in  Subsection  1  above  the  time  transition 

ij 

probabilities  for  the  imbedded  process  were  used  for  the  transition 
behavior. 

This  model  was  expected  to  be  superior  to  that  in  Subsection  1 
because  the  mean  waiting  time  in  state  i  was  now  made  dependent  on  the 
destination  state  j.  These  mean  times  were  given  in  Table  11.3.  The 
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h  (m)  =  P(transition  from  i  to  j  occurs  at  time  m  transition 
ij  1 


is  to  j  from  i) 


3 .  The  Second  Semi-Markov  Approximation:  Uniform  Distribution 


The  next  semi-Markov  approximation  assumed  a  uniform  distribu¬ 


tion  for  each  transition  time  T  .  The  lower  and  upper  limits  of  this 

ij 

distribution  were  selected  to  match  the  mean  and  variance  of  T  for  the 

ij 

true  process.  If  the  lower  and  upper  limits  are  denoted  by  ( a  ,b  ) — 

ij  ij 

the  mean  transition  time  by  T  and  the  variance  of  this  time  by  v  — the 

ij  ij 

relations  are 
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a  =  m  -  /"5v 
ij  ij  ij 


b  =  m  +  /3v 
ij  iJ  ij 


> 


where  a  is  replaced  by  zero  if  negative.  The  fit  obtained  with  this 

ij 

approximation  is  shown  in  Figure  11.4  and  seems  generally  superior  to 
that  obtained  by  matching  only  the  single  parameter  T 

ij 

4.  The  Third  Semi-Markov  Approximation:  Averaged  h^j 

The  third  semi-Markov  approximation  entailed  the  calculation  of 
density  functions  h. .(•)  by  appropriate  conditioning  methods  on  the  den¬ 
sity  functions  implied  by  the  PST  diagram.  As  suggested  by  the  notation, 
one  density  function  was  chosen  for  each  allowed  transition.  Because  of 
the  complexity  of  the  ways  some  of  the  densities  were  formed,  no  general 
pattern  or  form  of  density  could  be  imposed  in  all  cases  such  as  was  done 
in  the  previous  semi-Markov  model. 

A  comparison  of  the  true  and  estimated  P( state)  as  a  function 
of  time  curves  is  shown  in  Figure  11.5.  The  fit  is  somewhat  improved 
over  either  of  the  earlier  two  semi-Markov  approximations  but  is  still 
far  from  perfect.  The  reasons  for  the  imperfections  are  that  there  are 
correlations  in  the  true  process  that  are  not  reflected  in  the  semi-Markov 
model;  as  flexible  as  semi-Markov  models  are,  they  still  have  strong 
assumptions  of  independence  of  all  densities  h,  .(•)  for  each  transition. 


D.  Approximations  Using  Auxiliary  States 

Three  different  ways  to  add  auxiliary  states  for  improvement  in  the 
quality  of  the  P(state)  as  a  function  of  time  curves  were  formulated  and 
tested  numerically.  Some  improvement  was  noted  over  the  previous  methods 
that  used  the  original  state  space.  By  the  end  of  this  experimentation, 
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PISTATE(t)  SEARCHl 
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FIGURE  11.5  COMPARISON  OF  P (STATE)  AS  A  FUNCTION  OF  TIME  CURVES  FOR  ACTUAL 
PROCESS  AND  SEMI-MARKOV,  AVERAGED  h..  APPROXIMATION  (FIVE  STATES) 


the  nature  of  the  difficulty  with  the  actual  process  was  quite  clear; 
time  itself  would  have  to  enter  the  state  definition. 


1 .  The  Fourth  Semi-Markov  Approximation:  One-State  Memory 

A  state  transition  diagram  for  the  actual  process  that  effec¬ 
tively  "remembers"  the  previous  state  is  shown  in  Figure  11-6.  On  this 
diagram,  the  nodes  are  labeled  with  a  pair  of  integers — the  integer  to 
the  far  right  represents  the  current  state,  and  the  integer  to  the  left 
represents  the  last  state.  It  may  be  verified  that  the  states  of  this 
process  are  in  one  to  one  correspondence  with  the  branches  of  the  original 
process  whose  transition  diagram  is  shown  in  Figure  11.2.  There  are  now 
16  states  instead  of  nine  and  seven  absorbing  states  instead  of  two.  The 
probability  of  absorption  in  state  8  (fail)  is  now  the  sum  of  being  ab¬ 
sorbed  in  states  18,  28,  48,  58,  68,  and  78  since  there  are  six  states 
from  which  the  fail  state  can  be  entered. 


Transition  probabilities  are  again  shown  on  branches  in  the 
transition  diagram.  Several  of  the  transition  probabilities  are  identical 
to  those  in  the  original  process,  but  six  differ  as  a  result  of  the  con¬ 
ditioning  on  the  previous  state.  The  transition  probabilities  that  differ 
are  shown  with  superscripts  on  the  figure  together  with  their  values  in 
the  actual  process. 


A  new  formula  for  the  probability  of  being  absorbed  in  the  Kill 
state  is  required;  the  formula  is 


(1)  (1)  P13  P31P12  P25P56 

p=p  pp  p  p  „  +  - 

k  12  25  56  67  79 


(1)  (2)  (1)  (1)  (2) 

P13  P34P45P56  P67P79 


,  (2) 

P13  P31 


(The  first  term  has  no  denominator  since  the  open  path  corresponding  to 
the  numerator  does  not  touch  the  single  loop  in  the  diagram,  and  this 
makes  its  path  determinant  unity. ) 
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FIGURE  11.6  TRANSITION  DIAGRAM  AND  PROBABILITIES  FOR  ONE-STATE  MEMORY  MODEL 


A  comparison  between  the  true  probabilities  of  state  as  a  func¬ 
tion  of  time  and  those  predicted  by  the  semi-Markov  model  with  the  averaged 
h  (•)  calculated  for  the  states  in  Figure  11,6  is  shown  in  Figure  11.7. 

i  j 

The  fit  is  superior  to  any  other  obtained  for  states  search,  detect, 
classify,  and  kill  and  it  is  roughly  the  same  for  other  states.  The 
reason  for  the  lack  of  a  dramatic  improvement  is  that  the  deconditioning 
implied  by  going  to  a  one-state  memory  removed  some  dependencies,  but 
many  other  dependencies  were  unaffected. 

A  rough  measure  of  the  goodness  of  fit  of  the  predicted  curves 
to  the  actual  curves  was  adopted  so  that  the  various  fitting  methods  could 
be  compared  systematically.  The  results  tabulated  in  Table  11.8  show 
that  the  methods  gave  successively  better  results  for  all  states  except 
the  prosecute  false  contact  state.  Why  the  measures  do  not  decrease 
monotonically  for  this  single  state  is  unknown. 

The  last  row  of  Table  11.8  shows  the  measures  obtained  for  the 
direct  estimation  method.  The  probability  of  occupying  a  state  at  a  time 
t  was  estimated  simply  by  the  number  of  times  the  state  was  occupied  at 
time  t  divided  by  the  number  of  replications  (100).  Values  shown  in  the 
table  are  the  average  of  two  sets  of  100  Monte  Carlo  replications.  Al¬ 
though  the  sample  size  is  small  (two  sets),  this  simple  estimator  may  be 
superior  to  those  obtained  by  the  other  methods.  In  the  example  under 
study,  this  is  perhaps  to  be  expected  because  of  the  irregularities  in 
the  true  P(state)  as  a  function  of  time  curves,  and  the  direct  estimator 
is  able  to  "track'  these  irregular  curves  better  than  the  more  highly 
structured  Markov  and  semi-Markov  methods. 

2.  A  Loopless  Version  of  the  False  Contact  Model 

In  the  subsection  above,  a  systematic  method  was  used  to  in¬ 
crease  the  number  of  states  in  order  to  decondition  the  relationships 
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P]STAT£(t]  -  KILLl  P[STAT£]t)  =  CLASSIFY]  P[STAT£{tl  -  DETECT]  PISTATElt!  =  PROSECUTE  FALSE  CONTACT] 


Table  11.8 


MEASURES  OF  DYNAMIC  FIT 


Processes 

Search 

Kill 

Semi -Markov  Process 

Matched  mean  time  T 

ij 

109 

45 

47 

33 

28 

Match  mean  and 

variance  by  uniform 

distribution 

87 

55 

36 

28 

| 

24  | 

Averaged  h 

ij 

80 

64 

35 

24 

24 

One-state  memory 

46 

56 

28 

16 

17  i 

| 

Markov  Chain 

j 

100  replications 
(average  of  two 
cases) 

40 

22 

_ 

1 

] 

7 

Markov 

Least  squares 

87 

66 

43 

37 

61  : 

* 

The  measure  is: 


10 

constant  X  ^  P(state  =  k  at  time  20  n)  -  P(state  =  k  at  time  20  n) | 
n=l 


where  P( • )  is  actual  and  P(  •  )  is  predicted. 


somewhat  so  that  a  better  fit  to  the  P(state)  as  a  function  of  time  re¬ 
lations  might  be  attained.  In  that  model,  the  previous  state  was  "re¬ 
membered"  and  the  loop  between  the  (original)  states  search  and  prosecute 
false  contact  was  retained.  This  section  shows  how  the  loop  was  removed 
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by  the  addition  of  auxiliary  states  because  this  feature  of  the  model 
seemed  to  be  causing  problems. 

The  loop  is  removed  by  treating  each  new  entry  into  state  1  or 
state  3  as  entry  into  a  new  state.  When  the  notation  is  used  resembling 
the  one  in  the  model  that  remembered  the  previous  state,  the  state  labeled 
13  means  'the  state  is  3,  with  entry  from  state  l";  the  entry  131  means 
the  state  is  1,  with  entry  from  state  31."  In  general,  the  integers 
that  are  read  from  left  to  right  give  the  path  traversed  to  arrive  at  the 
state  represented  by  the  last  integer  on  the  right.  Figure  11.8  shows 
the  transition  diagram  and  the  true  values  of  the  transition  probabilities. 
It  can  be  noted  that  the  three  ways  of  entering  state  2  (opportunity)  from 
state  1  (detect)  have  different  transition  probabilities.  (The  branches 
have  probabilities  0.250,  0.185,  0.600.)  Similarly,  transitions  to  state 
8  (fail)  from  state  1  (search)  show  differing  probabilities.  (The 
branches  have  probabilities  0.100,  0.075,  0.445,  0.400.)  These  are  indi¬ 
cations  of  the  non-Markovian  nature  of  the  model  and  also  show  that  some 
"deconditioning"  is  accomplished  by  this  change  of  state  space.  The  de¬ 
rived  P( state)  as  a  function  of  time  curves  are  disappointingly  similar 
to  the  last  several  sets  and  are  therefore  not  plotted. 

3.  A  Semi-Markov  Model  with  Complete  Memory 

Considerable  effort  was  devoted  to  fitting  P( state)  as  a  func¬ 
tion  of  time  for  the  false  contact  example  before  it  was  realized  that  at 
least  part  of  the  difficulties  resulted  from  the  imbedded  process  not 
being  a  Markov  chain.  To  overcome  this  difficulty,  a  method  for  defining 
auxiliary  states  was  found,  the  new  state  space  automatically  provided 
a  complete  memory  of  earlier  states. 

The  PST  diagram  itself  can  be  used  to  define  the  new  state 
space.  Each  line  segment  on  the  PST  diagram  that  defines  a  state  change 
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can  be  identified  with  a  state  in  the  new  semi-Markov  process.  The 
labeling  of  the  new  states  is  shown  in  Figure  11.9;  there  are  47  states 
in  all.  The  transition  diagram  with  these  states  and  with  the  transition 
probabilities  computed  directly  from  the  PST  diagram  are  shown  in  Figure 

11.10. 

The  transition  diagram  has  a  single  path  for  each  separate  way 
of  reaching  an  absorbing  state  from  the  starting  state  (la).  Paths  in 
the  actual  process  are  therefore  in  1  to  1  correspondenpe  with  those  on 
the  transition  diagram.  The  probability  of  a  path  on  the  transition 
diagram  is  the  product  of  the  transition  probabilities  along  the  path, 
and  these  path  probabilities  agree  for  all  paths  in  the  actual  process 
and  in  the  semi-Markov  model  approximating  it  when  infinite  data  are 
available  to  estimate  the  transition  probabilities. 

Although  the  PST  diagram  was  used  here  to  generate  the  new  state 
space,  it  should  be  noted  that  observed  path  data  can  be  used  for  this 
purpose.  A  computer  program  can  be  written  to  generate  the  states  and 
to  do  the  necessary  transition  counting  in  terms  of  these  states  if 
desired. 

A  comparison  may  now  be  made  between  the  P(state)  as  a  function 
of  transition  number  n  for  the  actual  and  approximating  imbedded  Markov 
chains.  Table  11.9  shows  the  data.  The  fit  is  adequate  but  not  perfect 
because  of  the  presence  of  the  loop  in  the  imbedded  (actual)  process. 

After  the  imbedded  process  is  reasonably  well-approximated,  the 
dynamics  can  be  added.  A  uniform  distribution  that  represents  the  condi¬ 
tional  writing  time  is  associated  with  each  transition  (branch)  on  Figure 
11.10.  These  uniform  distributions,  with  parameters  taken  directly  from 
the  PST  diagram,  were  used  together  with  the  transition  probabilities  in 
a  semi-Markov  model  to  calculate  P(state)  as  a  function  of  time  curves  to 
compare  with  those  from  the  actual  process.  Figure  11.11  compares  the 
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FIGURE  11.9  NEW  STATE  DEFINITIONS  FOR  SEMI-MARKOV  MODEL  WITH 


COMPLETE  MEMORY 
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approximate;  0.629  =  actual 


P  j  ST  ATE  (n  I  PISTATE  =  PROSECUTE 

=  KILLl  FALSE  CONTACT)  PISTATE  =  OPPORTUNITY] 


TIME  - 


FIGURE  11.11  COMPARISON  OF  P(STATE)  AS  A  FUNCTION  OF  TIME  CURVES 
FOR  ACTUAL  PROCESS  AND  47  STATE  SEMI-MARKOV  PROCESS 
APPROXIMATION 
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results  for  the  search,  prosecute  false  contact,  and  kill  states.  The 
fit  appears  to  be  better  than  that  obtained  in  earlier  sections  because 
the  approximating  process  is  "responding”  for  the  first  time  to  some  of 
the  irregularities  in  the  actual  process.  The  curves  still  were  not  as 
good  as  expected.  After  more  experimentation,  the  principal  difficulty 
causing  all  approximations  to  be  poor  finally  became  evident;  the  prob¬ 
lem  can  be  seen  most  easily  on  the  PST  diagram. 

Consider  the  transition  from  state  la  to  state  3a  (0.40  to  1.0). 
Each  of  these  transitions  is  given  the  same  distribution  of  waiting  time 
in  state  la,  regardless  of  transitions  later  on.  It  is  apparent  from  the 
PST  diagram  that  this  wide  distribution  (transition  time  ranged  from  0  to 
60)  was  distorting  the  time  behavior  for  all  later  transitions.  For 
example,  the  transitions  to  the  kill  state  for  the  range  of  u  from  0.40  to 
0.42,  which  have  an  actual  la  to  3a  transition  time  varying  from  zero  to 
approximately  two  time  units,  have  transition  times  over  the  full  range 
from  0  to  60  time  units  in  the  47-state  semi-Markov  approximation.  This 
phenomenon,  repeated  in  many  places,  is  causing  the  poor  P( state)  as  a 
function  of  time  fits  for  all  of  the  approximations  considered.  It  is 
believed  that  time-in-state  itself  must  be  incorporated  into  the  state 
definition  in  order  to  overcome  this  difficulty.  Although  this  insight 
comes  from  the  form  of  the  PST  diagram  itself,  it  is  clear  that  analysis 
of  the  path  data  also  could  reveal  the  nature  of  this  obstacle  and  suggest 
a  way  to  remove  it. 

E.  Testing  the  Markov  Property 

Since  the  application  of  Markov  chain  models  is  being  considered  for 
a  variety  of  ASW  tactical  analyses,  it  is  of  interest  to  know  how  the 
hypothesis  of  "Markovianness"  can  be  tested.  When  the  time  behavior  of 
the  process  is  ignored  and  only  the  imbedded  Markov  chain  is  observed,  a 
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rather  simple  test  is  possible  when  data  are  available  on  entire  paths 
through  the  Markov  transition  diagram.  The  idea  is  that  when  a  Markov 
chain  with  known  transition  probabilities  is  hypothesized  the  path  proba¬ 
bilities  can  be'  calculated  from  the  formula 

P(path)  =  product  of  transition  probabilities  along  the  path  . 

To  obtain  this  relationship,  the  Markov  property  (lack  of  memory  at  a 
state)  is  invoked. 

Consider  a  Markov  chain  with  a  single  starting  state  and  two  absorbing 
states.  Let  the  paths  in  a  Markov  chain  be  labeled  according  to  some 
arbitrary  numbering  system — such  as  numbering  according  to  their  proba¬ 
bilities.  Since  there  are  infinitely  many  distinct  paths  possible  when¬ 
ever  the  transition  diagram  has  loops,  it  will  be  necessary  to  combine 
all  paths. beyond  some  number,  say  k  -  1.  Then  each  of  the  infinitely 

many  paths  falls  into  one  of  k  classes;  the  first  (k  -  1)  paths  each  has 

th 

its  own  class,  and  the  k  class  has  all  the  remaining  paths. 

The  classical  chi-squared  goodness  of  fit  test  may  now  be  applied. 

Let  there  be  n  independent  trials,  each  resulting  in  a  path  through  the 

system.  Let  p.  be  the  probability  of  observing  a  path  in  class  i 

(i  =  1,  2,  ...,  k)  and  f  be  the  observed  number  of  sample  paths  falling 

i 

in  class  i.  Then  according  to  Ref.  7  the  quantity 

2  y^(fi  ~  n  pi)2 

Xk-1  n  p. 

i=l  i 

is  asymptotically  distributed  as  chi-squared  with  k  -  1  degrees  of  freedom. 

In  the  false  contact  model  defined  by  the  PST  diagram  in  Figure  11.1, 
the  paths  and  their  proabilities  can  be  determined  by  straightforward  but 
tedious  calculations.  In  the  PST  diagram,  the  actual  probabilities  are 
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determined  by  finding  the  total  length  of  the  intervals  in  which  the  ran¬ 
dom  variable  u  determines  a  path  of  the  designated  type.  For  example, 
the  path  detect-fail  (designated  18)  is  generated  in  the  single  interval 
0  ^  u  <  0.10  so  that  P(path  18)  =  0.10. 

Table  11.10  compares  the  actual  path  probabilities  with  the  proba¬ 
bilities  that  are  derived  from  a  Markov  model.  Inspection  of  this  table 
shows  two  things  of  interest.  The  first  is  that  the  actual  path  probabili¬ 
ties  (which  reflect  the  Markov  assumption)  are  in  general  very  different 
from  the  probabilities  the  paths  must  have  if  the  process  is  truly  Markov 
with  transition  probabilities  obtained  from  infinite  sampling.  The  second 
aspect  of  interest  is  that  the  sum  of  the  path  probabilities  over  the  paths 
observable  from  the  actual  process  is  only  0.6177;  missing  paths  have 
probability  1  -  0.6177  =:  0.38.  Because  there  are  infinitely  many  paths 
with  positive  probability,  it  was  known  in  advance  that  the  sum  over  any 
finite  number  of  paths  would  be  less  than  unity  but  it  is  somewhat  sur¬ 
prising  to  see  the  sum  so  much  less  than  unity  in  this  example. 

Applying  the  chi-squared  test  to  this  example  using  data  from  Table 
11.10  will  always  result  in  failure  of  the  model  to  pass  the  test  whenever 
the  sample  size  is  around  100.  To  see  this,  consider  22  classes  into 
which  the  result  of  an  experiment  may  fall:  the  first  21  classes  corre¬ 
spond  to  the  21  possible  paths  enumerated  in  the  table  and  all  other 
possible  paths  comprise  the  remaining  class.  In  terms  of  the  chi-squared 
formula,  the  value  of  k  is  22,  and  the  last  term  of  the  sum  has  value 

(0  -  n  p  \^/n  p  =  n  p  =  expected  number  of  paths  in 

22/  *  22  22 

the  unobserved  class 
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Table  11.10 


COMPARISON  OF  PATH  PROBABILITIES 


Path 

Actual 

Probability  of  Path 

Markov 

Chain  Probability 

1 

8 

0. 10 

0. 1640 

1 

2 

8 

0.10 

0. 13565 

1 

2 

5 

8 

0.05 

0.031248 

1 

2 

5 

6 

8 

0.05 

0.038032 

1 

2 

5 

6 

7 

9 

0.10 

0.06581 

1 

3 

1 

3 

1 

3 

1 

2 

5 

6 

7 

9 

0.02 

0.0068272 

1 

3 

1 

3 

1 

3 

1 

2 

5 

6 

7 

8 

0.02 

0.001793 

1 

3 

1 

3 

1 

3 

1 

2 

8 

0.02 

0.014073 

1 

3 

1 

3 

1 

3 

1 

8 

0.04 

0.0011808 

1 

3 

1 

3 

1 

8 

0.12 

0.0096796 

1 

3 

1 

3 

1 

2 

5 

6 

8 

0.02 

0.007200 

1 

3 

1 

3 

1 

2 

8 

0.02 

0.0045591 

1 

3 

1 

3 

1 

2 

5 

8 

0.01 

0.00093462 

1 

3 

1 

2 

8 

0. 10 

0.059022 

1 

3 

4 

5 

6 

7 

9 

0.07 

0.001420 

1 

3 

4 

5 

6 

7 

8 

0.03 

0.000658 

1 

3 

4 

5 

6 

8 

0.04 

0.001449 

1 

3 

4 

5 

8 

0.03 

0.001190 

1 

3 

4 

8 

0.03 

0.016933 

1 

3 

1 

8 

0.03 

0.071358 

E  =  1.00 

E 

=  0.664045 
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If  there  are  100  samples  (n  =  100) ,  the  value  of  n  p  is 

100  X  (1  -  0.6177)  a  38  , 

which  corresponds  to  a  significance  level  of  0.99  at  k  -  1  =  21  degrees 
of  freedom.  The  remaining  terms  in  the  sum  are  all  nonnegative  so  that 
chi-squared  exceeds  38,  hence  the  hypothesis  is  rejected  at  the  0.99 
level.  Since  the  sums  of  squares  will  be  appreciable,  it  can  be  antici¬ 
pated  that  a  sample  size  of  much  less  than  100  will  practically  guarantee 
failure  of  the  test.  This  shows  the  process  to  be  so  badly  non-Markovian 
that  the  hypothesis  that  path  data  generated  by  the  process  could  have 
arisen  from  sampling  a  Markov  chain  is  always  rejected  for  samples  of 
size  100  or  greater.  The  chi-squared  test  thus  serves  a  useful  purpose 
in  this  example  by  alerting  the  analyst  to  the  strong  possibility  that 
the  model  to  which  Markov  methods  are  being  applied  is  actually  non- 
Markovian. 

Many  other  chi-squared  tests  also  can  be  devised  for  Markovianness 
of  the  process.  A  test  entailing  the  observed  proportions  of  the  time 
the  process  is  in  state  j  at  t  [y  ( t) ]  and  the  expected  proportions  p  ( t) 

j  ,  j 

is  readily  defined  by  noting  that  if  there  are  n  samples  of  the  complete 

process,  then  n  p,(t)  is  the  expected  number  of  samples  for  which  the 

state  is  j  at  time  t  and  n  y,(t)  is  the  observed  number  at  time  t.  When 

2 

all  times  and  all  r  states  are  summed,  a  random  variable  x  that  is 

7  (r-l)T 

asymptotically  chi-squared  with  (r  -  1)T  degrees  of  freedom  results  when 
the  process  is  Markovian. 

„  _L  fn  y.ft)  -  n  p.^)]2  «.r  nfv  (t)  “  p-(t)l~ 

2  _  L  j _ j  -i  _  L  j _ j  j 

X(r-1)T  /  ^  /  j  n  p.(t)  /  j  P.(t) 

t=l  j=l  J  t=l  j=l  J 

This  formulation  assumes  that  all  p  (t)  are  nonzero.  Practical  experience 

j 

with  the  chi-squared  test  suggests  that  the  classes  into  which  observations 
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may  fall  should  be  such  that  at  least  five  observations  are  expected  in 
each  class.  These  considerations  must  be  considered  in  the  use  of  this 
test,  and  some  alteration  of  the  summation  region  and  an  adjustment  in 
the  number  of  degrees  of  freedom  will  result. 
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XII  MARKOV  MODELS  AND  MONTE  CARLO  SIMULATION 

A .  Introduction 

This  section  shows  how  Markov  models  can  be  used  in  conjunction  with 
a  Monte  Carlo  simulation  model  for  one  of  several  purposes.  One  reason 
to  consider  a  Markov  model  in  a  context  in  which  a  simulation  model 
exists  may  be  simply  to  explain  and  interpret  the  simulation  model: 
suitably  choosing  a  state  space  and  recording  transition  counts  and 
transition  times  may  make  it  possible  to  gain  insight  into  the  simulated 
problem  that  could  not  be  gained  by  examination  of  the  end  results  of  the 
simulation  alone.  A  second  reason  may  be  to  build  an  auxiliary  model 
that  is  easier  to  use  for  some  purposes  other  than  the  simulation  itself, 
such  as  sensitivity  analysis.  In  some  situations,  it  may  be  possible  to 
replace  the  simulation  model  entirely  by  the  Markov  model  once  the  simu¬ 
lated  system  is  understood.  A  third  reason  that  a  Markov  model  may  be 
desirable  relates  to  validation  of  the  simulation.  Because  the  P(state) 
as  a  function  of  time  relationships  effectively  characterizes  the  model 
dynamics  and  a  valid  Monte  Carlo  model  should  adequately  model  the 
dynamics  of  the  tactical  situation  as  well  as  the  end  outcomes,  the  simu¬ 
lation  may  be  validated  by  a  comparison  of  the  P(state)  as  a  function  of 
time  curves  that  it  generates  with  the  same  relationships  estimated  from 
operational  data. 

In  this  section  some  experimentation  has  been  done  using  data  from 
a  Monte  Carlo  simulation  model  developed  by  NWRC  for  studying  the  effec¬ 
tiveness  of  acoustic  deception  devices  (ADDs)  in  a  transit  area  scenario. 
The  simulation  data  were  generated  to: 
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•  Determine  whether  the  simulation  model  could  be  adequately 
represented  by  a  Markov  model. 

•  Estimate  the  parameters  of  the  Markov  model. 

•  Determine  properties  of  the  Markov  model  that  could  be  com¬ 
pared  with  the  corresponding  properties  of  the  simulation. 

•  Show  how  some  sensitivity  analysis  may  be  performed  by  using 
the  Markov  model  with  its  estimated  parameters. 

B.  The  Simulation  Model 

The  simulation  model  selected  is  described  in  detail  in  Ref.  10. 

In  outline,  a  single  submarine  and  a  single  High  Value  Unit  (HVU)  are 
contained  within  an  objective  area  represented  by  a  circle  with  radius 
200  nmi.  The  HVU  or  its  supporting  forces  may  deploy  various  numbers 
of  acoustic  deception  devices  so  that  submarine  detection  and  classifica¬ 
tion  of  the  HVU  will  be  more  difficult.  The  submarine  searches  until  it 
detects  either  the  HVU  or  an  ADD;  detecting  an  ADD  "captures"  the  sub¬ 
marine  for  a  period  of  time  called  "capture  time."  During  the  capture 
time,  the  submarine  closes  the  ADD  to  investigate  it  further  and  then 
returns  to  search  at  the  end  of  the  capture  time  period.  All  units  have 
prescribed  forms  of  random  motion  consisting  of  straightline  tracks  with 
random  course  changes  superimposed.  Capture  may  occur  any  number  of  times 
before  detection  of  the  HVU  occurs,  and  the  model  stops  when  the  HVU  has 
been  detected.  Principal  inputs  to  the  model  are  speeds  of  the  HVU, 
submarine,  ADD,  length  of  the  straightline  track  between  random  course 
changes,  sonar  detection  ranges  of  the  HVU  and  the  ADDs,  and  a  parameter 
essentially  describing  the  length  of  time  that  a  submarine  "remembers"  a 
previously  detected  ADD,  Parameters  selected  correspond  to  those  shown 
on  Figure  5-2  of  Ref.  10;  they  are: 

Number  of  ADDs  =  1 
Capture  time  =  20  hours 
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Detection  ranges  (on  both  the  HVU  and  ADD)  =  60  nrai 

Memory  time  =  12  hours 

Speed  of  all  units  =  10  knots 

Length  of  straightline  track  segments  =  50  nmi, 

C.  Selection  of  States  for  the  Markov  Model 

The  states  originally  selected  were  those  used  in  an  analytical 
semi-Markov  model  of  the  same  scenario  as  described  in  Ref.  6.  States 
in  this  three-state  model  were  designated  search,  capture,  and  detection 
(of  the  HVU),  with  detection  being  an  absorbing  state  and  search  the 
starting  state.  However,  examination  of  the  data  soon  revealed  that 
there  should  be  two  search  states — initial  search  and  search — because  of 
the  considerable  difference  in  the  parameters  for  initial  searches  and 
later  searches.  The  transition  diagram  for  the  imbedded  Markov  chain  is 
shown  in  Figure  12.1.  (See  also  Appendix  F. ) 

START 


4 

FIGURE  12.1  IMBEDDED  MARKOV  CHAIN  FOR  SIMULATION  MODEL 

Since  the  simulation  model  did  not  employ  Markovian  assumptions,  it 
was  decided  that  a  test  of  the  reasonableness  of  a  Markov  approximation 
should  be  performed.  This  was  accomplished  by  the  same  method  used  to 
obtain  a  loopless  version  of  the  false  contact  model  in  Section  XI. 
Basically,  the  loop  in  Figure  12.1  is  unwound  by  considering  successive 
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entries  into  capture  and  search  as  new  states  as  shown  in  Figure  12.2. 
For  convenience,  the  states  are  numbered  as  shown. 

In  a  Markov  model,  the  transition  times  between  states  are  statis¬ 
tically  independent  of  one  another.  If  the  simulation  model  is  Markov, 
it  should  then  exhibit  this  property  also;  therefore,  a  crude  test  of 
Markovianness  can  be  formulated  on  the  basis  of  this  assumption. 

Let  T  be  the  average  time  that  state  i  is  first  entered,  and 
T  =  mean  time  spent  in  state  i,  given  that  the  next  transition  is  to 

ij 

state  j.  Observing  that  each  capture-to-search  transition  should  have 

the  same  mean  time  T  and  each  search- to-capture  transition  should  have 

c 

the  same  mean  time  T  ,  independence  of  transitions  implies 


T 


3 


+  T 

c 


T  =  T  +  T  +  T 
4  12  c  s 


T  =  T  +T+T+T=T  +2T+T 
5  12  c  s  c  12  c  s 


T 


6 


T  +  2T 
12  c 


4-  2T 

s 


Inspecting  T  for  even  n  shows  that  the  quantity  (T  +  T  )  is  added 
n  c  s 

to  T  for  each  successive  capture-search-capture  cycle.  Therefore, 
n-2 

the  even  values  of  T  should  plot  in  a  straight  line  as  a  function  of  n. 

n 

Figure  12.3  shows  a  plot  of  T  as  a  function  of  n  computed  from  1000 


replications  of  the  simulation.  The  plotted  values  for  even-numbered 
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300 


FIGURE  12.3  MEAN  TIMES  OF  FIRST  ENTRY  INTO  STATE  i  (T) 


states  fall  very  nearly  on  a  straight  line,  which  tends  to  confirm  the 
assumption  of  independence  of  transition  times.  A  straight  line  has  been 
drawn  through  the  initial  point  to  represent  a  mean  search  time  of  28.9 
hours  and  mean  capture  time  of  20  hours  (mean  search  time  was  calculated 
from  the  same  set  of  replications).  Other  sets  of  replications  also  gave 


this  same  near-linear  pattern  and  agreement  between  the  T  data  points 

i 

and  the  continuous  line  found  by  averaging  search  times. 

The  complete  output  from  the  set  of  1000  replications  is  shown  in 

tabular  form  in  Table  12.1.  T  and  T  have  already  been  defined, 

i  i , i+1 

var  T  is  the  variance  of  T  ,  and  var  T  is  the  variance  of  T 

i  i  i,i+l 

The  number  of  replications  entering  state  i  is  N  .  and  the  values  decrease 

i 

because  of  detections  of  the  HVU.  The  last  column  is  the  estimate  of  the 

transition  probability  p  obtained  from  the  ratio  N  /N  . 

i,i+l  i+1  i 

Table  12.1 

COMPUTER  SIMULATION  OUTPUTS 


State 

State 

Number 

T 

i 

var  T 

i 

sn 

Bin 

N 

i 

EH9 

S 

1 

0 

0 

0 

0 

1000 

0.473 

C 

2 

89.5 

9,568 

89.5 

9568 

473 

0.922 

S 

3 

109.5 

9,843 

0 

436 

0.706 

C 

4 

134.8 

11,952 

27.0 

2030 

308 

0.948 

S 

5 

152.0 

11,891 

0 

292 

0.733 

C 

6 

182.0 

15,118 

3539 

214 

0.953 

s 

7 

200.5 

14,752 

20.0 

0 

204 

0.819 

c 

8 

224,5 

16,674 

30.9 

3009 

167 

0.940 

s 

9 

245.4 

17,296 

20.0 

0 

157 

— 

Mean  times  to  first  reach  state  i  should  also  be  obtainable  by  accu¬ 
mulating  the  values  in  the  T  column  as  shown  in  Table  12.2,  where 

i,i+l 

the  T  are  repeated  for  comparison, 
i 

The  two  estimates  for  T  do  not  agree  because  they  are  over  different 

i 

samples;  the  T.  are  averaged  over  only  those  cases  reaching  state  i,  while 
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Table  12.2 


COMPARING  MEANS  AND  VARIANCES 


State 

Number 

*i 

£T 

i ,  i+1 

var  T 

i 

n-1 

7  var  T 

Z— {  i,  i+1 

i  =  l 

1 

0 

o 

0 

0 

2 

89.5 

in 

05 

00 

9,568 

9,568 

3 

109.5 

109.5 

9,843 

9,568 

4 

134.8 

136.5 

11,952 

11,598 

5 

152.0 

156.5 

11,891 

11,598 

6 

182.0 

186.6 

15,118 

15,137 

7 

200.5 

206.6 

14,752 

15,137 

8 

224.5 

237.5 

16,674 

18,146 

9 

245.4 

257.0 

17,296 

18,146 

the  are  obtained  by  averaging  over  all  transitions  between  i  and 

i  +  1.  (Since  more  data  are  used  to  estimate  the  values  in  the  second 
column,  their  variances  should  be  somewhat  smaller.) 

Several  other  tests  of  the  reasonableness  of  the  Markov  assumption 

can  be  made  from  the  data  in  Table  12.1.  Since  the  random  variables 

T  are  assumed  to  be  independent,  their  variances  should  also  be 

i, i+1  ' 

additive;  that  is, 

n-1 

var ( T  j  =  7  var  T 

\  n/  X— J  i, i+1 

i=l 

The  columns  on  the  right  side  of  Table  12.2  compare  the  two  estimates  of 

var  T  but  they  do  not  agree  for  the  same  reasons  that  the  means  do  not 
i 

agree,  A  rough  estimate  of  the  average  variance  per  cycle  is  found  by 

averaging  the  last  three  values  in  the  var  T  column;  the  average 

i ,  i+1 
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variance  is  (2030  4-  3539  +  3009)/3  =:  2859  and  the  average  standard  devia¬ 
tion  is  therefore  /"2859  e:  53.5. 

If  the  waiting  time  in  the  search  state  were  exponentially  distributed, 
this  standard  deviation  would  be  expected  to  be  about  equal  to  the  mean 
value  of  28.9  hours.  The  disparity  between  these  values  indicates  that 
the  waiting  time  densities  are  not  geometric  as  they  are  in  a  Markov  chain. 
(Despite  this,  a  Markov  chain  approximation  works  quite  well.) 

D.  Estimating  the  Transition  Probabilities  from  Transition  Data 

On  the  basis  of  the  p.  .  ^  column  in  Table  12.1,  the  alternating 
values  0.922,  0.948,  0.953,  and  0.940  should  all  be  estimates  of 

P[S(n  +  1)  =  search|S(n)  =  capture]  =  p  =  p 

S/ C 

The  values  cannot  be  averaged  to  obtain  p  because  different  sample 
sizes  are  entailed.  Similarly,  the  alternating  values  0.706,  0.733,  and 
0.819  all  estimate 

P[S(n  +  1)  =  capturel S(n)  =  search]  =  p  =  p  . 

1  32  c/s 

The  value  at  the  top  of  the  column  (0.473)  is  an  estimate  of 
P(next  state  is  capture] this  state  is  initial  search)  =  p 

X  z 

From  symmetry  of  the  HVU  and  ADD  with  respect  to  the  submarine's  Initial 
Search,  this  value  is  expected  to  be  close  to  1/2. 

The  weighted  estimates  of  p  ,  and  p  may  be  found  most  easily  by 

s/c  c/s 

using  the  N.  column  of  Table  12.1. 
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„  _  436  +  292  +  204  +  157 

Ps/c  “  473  +  308  +  214  +  167 


0.9372 


*  308  +  214  +  167 

p  ,  =  -  2:  0.7393 

c/s  436  4-  292  +  204 


Therefore,  the  transition  diagram  for  the  imbedded  chain  is  that 
shown  in  Figure  12.4. 


.7393 


FIGURE  12.4  TRANSITION  DIAGRAM  FOR  IMBEDDED  CHAIN  OF  SIMULATION  MODEL 

From  here  it  is  a  short  step  to  the  Markov  chain  approximation. 

Weighting  the  average  times  T  by  the  N.  and  normalizing  give  an  average 

duration  of  search  of  28.9  hours.  Therefore,  the  self-loop  p  should 

have  a  value  1  -  1/28.9  =  0.9654  to  give  the  correct  average  waiting 

time  in  state  3  (search).  The  p  that  gives  an  average  of  89.5  hours 

in  state  1  is  1  -  1/89.5  =  0,9888.  Similarly,  since  the  average  time  in 

the  capture  state  (2)  is  the  constant  capture  time  of  20  hours,  the  value 

of  p  is  1  -  1/20  =  0.95.  Normalizing  the  probabilities  shown  on  Figure 
2  2 

12.4  for  the  imbedded  chain  results  in  the  transition  diagram  for  the 
Markov  chain  approximation  as  shown  in  Figure  12.5. 

With  the  transition  probability  estimates  shown  in  Figure  12.5,  the 
P(state)  as  a  function  of  time  estimates  can  be  found  by  raising  the 
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FIGURE  12.5  TRANSITION  DIAGRAM  FOR  MARKOV  CHAIN  APPROXIMATION 

transition  matrix  P  to  successive  powers.  Figure  12.6  shows  the  following 
principal  curve  of  interest: 

P  (t)  =  P(HVU  is  not  detected  by  time  t)  =  1  -  P(absorption  in 

state  4  by  time  t) 


Also  shown  on  Figure  12.6  is  another  estimate  of  this  probability  ob¬ 
tained  from  the  ratios 


Pg(t)  =  P(HVU  is  not  detected  by  t) 


number  of  replications  for 
which  HVU  is  not  detected  at  t 
number  of  replications 


A 

[P  (t)  was  calculated  from  an  independent  set  of  300  replications,  the 
s 

A 

1000  replication  set  used  to  obtain  P  (t)  could  not  be  used  due  to  com- 

M 

puter  limitations.] 

The  correspondence  between  p  (t)  and  p  (t)  is  quite  good.  It  shows 

M  s 

that  the  Markov  chain  approximation  may  be  adequate  even  though  the 
assumption  of  exponential  holding  times  is  not  satisfied  at  the  capture 
and  search  states. 
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FIGURE  12.6  COMPARISON  OF  P(NO  DETECTION  BY  TIME  t)  FOR  SIMULATION 
MODEL  AND  MARKOV  CHAIN  WITH  PARAMETERS  ESTIMATED 
FROM  1000  INDEPENDENT  TRIALS 


The  average  time- in-state  (transmission)  matrix  is  easily  found  to 


be 


T  = 


(I  -  Q) 


-1 


(89.2857 
° 

0 


30.9302  41.9256 
65.3619  88.5976 
48.3603  94.4537 


The  (1,1)  element  is  the  average  duration  of  the  initial  search,  and 
it  agrees  quite  well  with  the  value  89.5  hours  shown  in  Table  12.2.  The 
average  time  in  the  capture  state  is  the  (1,2)  element  (30.9302  hours). 
Therefore,  an  estimate  of  the  average  number  of  captures  per  trial  is 

_ average  time  in  capture _  30.9302  ^ 

average  duration  of  a  single  capture  20 

Since  there  is  only  a  single  absorbing  state,  the  absorption  matrix 
A  =  (I  -  Q)  must  have  all  components  equal  to  one;  this  is  easily  veri¬ 
fied  numerically.  The  average  time  till  detection  matrix  is 

(I  -  Q)_1( I  -  Q)_1R  =  TA  , 


and  no  normalization  is  necessary  because  of  the  single  absorbing  state. 
Using  values  from  Figure  12.5,  we  obtain 


/162. 142\ 
TA  =  I  153.959  1 
\ 142 . 814/ 


(mean  time  to  Detect  starting  in  state  1 

mean  time  to  Detect  starting  in  state  2 

mean  time  to  Detect  starting  in  state  3 


so  that  the  mean  time  to  detection  of  the  HVU  is  about  162  hours.  (The 
sum  of  the  elements  in  the  first  row  is  also  162.142  hours  because  there 
is  only  one  absorbing  state.) 
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It  is  informative  to  consider  the  individual  terms  of  the  inner 


product  of  the  first  row  of  (I  -  Q)  and  R,  for  these  terms  are  the 
average  numbers  of  detections  from  states  1,  2,  and  3,  respectively. 
Thus, 


Average  number  of  HVU 

detections  from  initial  search  =  89.2857  X  0.0059  =  0.527 

Average  number  of 

detections  from  capture  =  30.9302  x  0.0031  S  0.096 

Average  number  of 

detections  from  search  =  41.9256  x  0.0090  =:  0.377 

(The  sum  of  these  elements  is  unity,  the  probability  of  absorbing  in 
state  4.)  Although  there  are  1.5  captures  on  the  average,  slightly  more 
than  half  of  the  HVU  detections  occurred  without  a  capture.  The  average 
number  of  HVU  detections  from  initial  search  (0.527)  is  seen  to  agree 
with  the  l-to-4  transition  probability  for  the  imbedded  chain  of  Figure 
12.4. 

E.  Estimating  the  Parameters  from  Average  Time-in-State  Data 

Section  VI-C-2  showed  that  estimates  t . .  of  the  average  time  spent 
in  each  state  j  (given  starting  state  i)  can  be  used  to  estimate  transi¬ 
tion  probabilities.  A  set  of  50  replications  was  run  to  obtain  the  data 
shown  in  Table  12.3.  All  data  were  generated  starting  in  state  1;  how¬ 
ever,  and  the  t  must  be  obtained  for  starting  states  2  and  3  also.  Re- 
ij 

moving  the  transitions  from  state  1  to  state  2  made  it  possible  to  estimate 
the  t  .  with  data  from  the  same  set  of  replications;  similarly,  the  t 
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Table  12,3 


SIMULATION  OUTPUT  WHEN  USING  AVERAGE  TIME-IN-STATE  DATA 


State 

HUB 

N 

i 

1 

0 

50 

2 

76.9 

27 

3 

20.0 

26 

4 

16.1 

24 

5 

20.0 

22 

6 

36.8 

15 

7 

20.0 

13 

8 

15.6 

11 

9 

20.0 

8 

10 

21. 1 

1 

11 

20.0 

H 

12 

12.5 

13 

20.0 

14 

18.5 

B 

were  estimated  by  removing  all  transitions  before  the  first  entry  into 
state  3. 

The  estimated  transition  matrix  ¥  of  average  times-in-state  is 


/76 .9 
T=  0 
\  0 


34  33. 6\ 

63  62,2  1  =  (I  -  Q) 

44.6  64.6/ 


from  which  Q  is  determined  to  be 
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0.986996 

0 

0 


0,006939 

0.950288 

0.034422 


0.000111 

0.047801 

0.951376 


Q  =  (I  -  T)"1 


The  R  is  determined  so  that  the  sum  of  the  elements  in  each  row  of  P  is 
unity.  Thus,  the  estimated  transition  matrix  P  is 


P  = 


(3 

R)  = 

\o 

0.986996 

0. 006939 

0.000111 

0.005954 

0 

0.950288 

0.047801 

0.001911 

0 

0,034422 

0.951376 

0.014202 

0 

0 

0 

1  t 

~  -1~  T 

[A  calculation  can  be  made  to  see  whether  ( I  -  Q)  R  equals  (1,1,1)  ;  the 
,  T 

result  is  (1.0000238,  1.0037574,  1.0026789)  which  is  satisfactorily 
close. ] 

Raising  P  to  the  nth  power  and  plotting  1  -  pj^  as  a  function  of 
time  n  produces  the  results  in  Figure  12.7. 

Another  estimate  of  this  same  curve  for  P(no  detection  by  time  n) 
was  made  by  using  the  transition  data  and  average  waiting  time  data  as 
in  the  earlier  1000  replication  case.  The  estimated  transition  matrix 
P  is 


P  = 


0.0070 

0.95 

0.0357 

0 


0 

0.0441 

0.9539 

0 


( n) 

When  P  is  used  to  determine  1  -  p  =  P(no  detection  of  HVU  by  n), 

the  curve  plots  essentially  on  top  of  the  P-derived  curve  just  found  as 
shown  in  Figure  12,7.  In  this  case,  the  two  methods  of  estimating 
P(no  detection  of  the  HVU  by  time  n)  appear  to  be  virtually  equivalent. 
(Whether  this  will  also  be  true  in  other  contexts  is  unknown.  It  is  easy 
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FIGURE  12.7  COMPARISON  OF  P(NO  DETECTION  BY  TIME  t)  FOR  TRANSITION 
PROBABILITIES  ESTIMATED  FROM  TRANSITION  DATA  AND 
AVERAGE  TIME-IN-STATE  DATA 


to  show  by  examples  that  the  p  and  p 

ij  ij 


from  the  same  data.) 


are  not  identical  when  estimated 


F.  Average  Time  Until  Detection  of  the  High  Value  Unit 


In  both  the  simulation  model  and  Markov  model,  the  HVU  is  ultimately 
detected  by  the  submarine.  Therefore,  the  mean  time  until  detection  is 
implicitly  conditioned  on  detection  ultimately  occurring.  Since  the 
starting  state  is  1,  the  Markov  chain  model  gives 


T  =  mean  time  until  detection 
D 


=  sum  of  elements  of  first  row  of  (I  -  Q) 


(12.1) 


A  more  general  formula  for  mean  time  until  detection  can  be  readily 

derived  for  a  semi-Markov  model.  The  mean  time  to  detect  depends  only 

on  the  mean  times  of  the  transitions  (T  )  and  the  transition  probabili- 

i  j 

ties  p  for  the  imbedded  chain.  (This  can  be  seen  by  conditioning  on 
ij 

the  path  taken  and  noting  that  the  mean  time  for  each  path  is  the  sum  of 

the  mean  transition  times  along  that  path.)  Therefore,  the  transition 

time  from  i  to  j  can  be  considered  to  be  concentrated  at  the  mean,  re- 

T.  . 

suiting  in  a  z  transform  of  the  form  p  z  1J  on  the  (i,j)  branch. 

ij 

Dropping  all  the  "bars"  over  the  mean  times  T  is  typographically 

ij 

convenient.  Figure  12.8  shows  the  transition  diagram  and  z  transforms 
of  the  transition  time  densities. 

The  transmission  g(z)  from  state  1  to  state  4  is  required  to  calcu¬ 
late  T  . 

D 
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12/ 


24 


,  ,  14  *12 

g(z)  =  P14z  +  - 


p  z  I  p  z  +ppz 

“in  \  “n  /i  “  n  ii 


T  +T  \ 
23  34' 


24 


23  24 


1  -  p  p  z 
P23P32 


T  +T 
23  32 


.  (12.2) 


Differentiating  with  respect  to  z,  setting  z  =  1,  and  rearranging  gives 
the  average  time  until  detection  T 


P  P 
12  24 

T  =  g'(z  =  1)  =  p  T  +pT  +  - 

D  b  14  14  12  12  1  -  p 


p 

23  32 


P  P 
12  23 

T  „  +  - 

24  1  -  p 


23P32 


23 


P  P  P  P  P  P 

12  23  32  12  23  34 

+  -  T  +  -  T 

1-pp  32  1-pp  34 

“ nn“ on  rno  rr  - 


(12.3) 
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FIGURE  12.8  SEMI-MARKOV  MODEL  WITH  ALL  TRANSITIONS  AT  THEIR 
AVERAGE  TIMES 


The  multipliers  of  the  average  times  T  are  easily  interpreted 

i  j 

here.  The  coefficient  of  T  is  p^,  the  average  number  of  times  that 
the  1  ->  4  transition  is  made.  Similarly,  p  (the  coefficient  of  T  ) 
is  the  average  number  of  times  that  the  1  -*  2  transition  is  made.  In 
general,  the  coefficient  of  T  is  the  average  number  of  i,j  transitions. 

ij 

Therefore,  the  average  time  until  detection  is  a  sum  over  all  transitions 
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i  ->  j,  of  products  of  the  form 


(average  number  of  i  ->  j  transitions)  X  (average  time  required  for  an 


j  transition) 


Differentiating  the  T  formula  of  Eq.  (12.3)  with  respect  to  the  T 

D  _  ij 

gains  some  insight  into  the  sensitivity  of  to  the  T.  ..  Of  special 

interest  are  the  average  capture  time  (given  a  return  to  search)  T  and 

23 

average  search  time  (given  a  return  to  capture)  T  .  We  have 


ST 


ST 


23 


P  P 
12  23 

1  -  p  p 

*23  32 


(12.4) 


STD  P12P23P34 


ST 


34 


1  P23P32 


(12.5) 


Thus,  P10P„/(1  “  is  the  increase  in  the  average  time  until  de- 

tection  (T^)  per  hour  increase  in  average  capture  time  (T  )  for  small 

changes  in  T  ;  all  other  parameters  remain  constant.  Similarly,  a  one 

hour  increase  in  T  induces  an  increase  ofp  p  p  /(1-p  p  )  hours 
34  12  23  34  23  32 

in  T  •  all  other  parameters  remaining  constant.  Using  the  p  found  for 

D  ij 

the  1000  replication  case  (Figure  12.4)  gives  estimates 

ST_ 


D 


ST 


s  1.45  hours/hour 


23 


ST 


D 


ST 


=  0.3 


hours/hour 


34 


In  the  simulation  model,  the  Capture  -*  Detect  HVU  transition  occurs 
at  the  end  of  capture  if  the  HVU  is  within  detection  range.  Therefore, 
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T  =  T  so  that  the  T  term  in  Eq.  (12.3)  becomes 

£  i—d  iJ 


P12(P24  +  P23 ) 


1  -  p  p 

f23f32 


23 


P  T 
12  23 

1  -  p  p 

23  32 


(12.6) 


which  equals  the  average  number  of  times  the  submarine  is  captured. 

sensitivity  analysis  then  simply  shows  that  each  hour  increase  in  T 

implies  an  increase  of  p  /(I  -  p  p  )  =  average  number  of  captures 

12  2332 

For  this  example ,  the  average  number  of  captures  is  1.54  so  that  each 
hour  increase  in  average  capture  time  increases  by  this  amount. 


The 
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XIII  COMBINING  PARAMETER  ESTIMATES 


A.  Introduction 

Section  IV  asserted  that  Markov  models  have  a  convenient  structure 
for  bringing  together  parameter  estimates  obtained  from  various  sources 
into  a  single  effectiveness  model.  When  more  than  a  single  estimate  of 
a  parameter  are  available,  some  method  is  required  to  combine  the  separate 
estimates  into  an  overall  parameter  estimate.  This  section  shows  how 
estimates  may  be  combined  in  an  optimum  linear  way  when  the  estimates  are 
statistically  independent  and  their  variances  are  known. 

Other  topics  in  this  section  deal  with  variance  considerations  in 
Markov  chain  models  when  the  probability  of  success  is  the  measure  of 
interest.  Variance  calculations  are  first  reviewed  for  a  simple  Bernoulli 
model  and  then  extended  for  a  more  complex  model  that  has  some  relevance 
for  exercise  design.  The  section  concludes  with  a  model  for  determining 
the  amount  of  effort  that  should  be  devoted  to  sampling  at  a  state  in  a 
Markov  chain  model  of  a  simple  situation  when  the  costs  of  sampling  are 
considered.  Although  the  model  is  too  simple  to  be  realistic,  it  illus¬ 
trates  how  the  available  effort  may  be  theoretically  apportioned  by  de¬ 
signing  exercises  to  minimize  the  variance  of  the  estimator  for  the 
probability  of  success. 

B.  Combining  Independent,  Unbiased  Estimates 

Suppose  and  X^  are  independent  estimates  of  the  same  quantity  M, 

and  that  each  is  unbiased  with  common  mean  M.  Variances  of  these  random 

variables  are  assumed  to  be  different,  however,  and  they  are  denoted  by 
2  2 

a  and  cr  ,  respectively. 

1  2 


229 


Elementary  calculus  can  be  used  to  combine  these  two  estimates  in 

the  best  linear  way.  Specifically,  if  the  improved  estimate  Y  is  defined 

as  a  linear  combination  of  X  and  X  by 

1  2  J 

Y  =  aX  +  <1  -  a)X  (13.1) 

1  2 


for  some  fixed  nonnegative  a,  then  a  can  be  found  to  make  the  variance 
of  Y  a  minimum.  For  any  a,  Y  is  an  unbiased  estimate  of  M,  because 


E(Y) 


E^aX^  +  (1  - 


=  aM  +  (1  -  a)M  =  M 


(13.2) 


Since  a  is  a  constant  and  X  ,  X 

1'  2 


2 

var  Y  =  a  var 


are  independent,  the  variance  of  Y  is 

2 

X  +  (1  -  a)  var  X 
1  2 


2  2  2  2 
a  a ,  +  (1  -  a)  a 


(13.3) 


Differentiating  Eq.  (13.3)  with  respect  to  a,  setting  the  derivative 
equal  to  zero,  and  solving  for  a  gives  the  optimal  weights  a  and  (1  -  a) 


a 


2 

a 

2 


2  2 

a  +  a 

1  2 


2 

a 


a  +  a 

1  2 


(13.4) 


That  is,  an  estimate  is  weighted  in  proportion  to  the  variance  of  the 
other  estimate  when  there  are  two  variables.  From  Eqs.  (13.1)  and  (13.4) 
the  weighting  formula  is 


Y 


2 

a  +a 

1  2 


-  X  + 

2  1 


2 

a  +  a 

1  2 


X 
2  2 


(13.5) 
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The  variance  of  Y  for  this  value  of  a  is  found  from  Eqs.  (13.3)  and  (13.4) 


var  Y 


2  2 
0  O 
1  2 

2  2 

0+0 
1  2 


(13.6) 


2  2 

Without  loss  of  generality,  we  may  assume  that  a  <  0  ,  so  that 


2  2  2  2 

0  0  0  O 

1  12  2 

—  <  var  Y  =  - <  — 

2  2  2  2 
0+0 
1  2 


(13.7) 


z 

If  0  is  arbitrarily  fixed  at  unity,  then  the  resultant  variance  of  Y  may 

2 

be  plotted  versus  o^: 


VARIANCE  Y 


VARIANCE  Y  = 


The  diagram  shows  that: 

•  The  variance  of  the  combined  estimate  is  always  less  than 
the  variance  of  the  best  estimator. 

•  The  overall  variance  is  halved  when  variances  are  equal. 

The  above  derivation  can  easily  be  generalized  to  three  statistically 

independent  estimators  of  the  same  quantity  M.  If  X  ,  X  ,  X  are  estimates 

12  3 

2  2  2 

with  mean  M  and  variances  a  a  a  then  the  appropriate  linear  combina- 
tion  is 


=  aX^  +  bX^  +  (1  -  a  -  b)X^  for  1  S  a,  b  ^  0  .  (13.8) 
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The  optimal  weights  turn  out  to  be 


a  =  a2a2/D 
2  3 


b  =  a2a2/D 
1  3 


2  2 

l-a-b  =  acr/D 
1  2 


(13.9) 


2  2  2  2  2  2 

where  D  is  the  normalizing  constant  (cr^cr^  +  ai°3  Thus>  a  weight 

is  the  product  of  the  other  pair  of  weights,  normalized.  (The  generaliza¬ 
tion  to  four  or  more  estimates  will  not  be  done  here.  The  formula  may 
entail  more  complex  functions  than  might  be  guessed  from  extrapolating 
the  cases  above. ) 

C.  Application  to  Estimation  of  Transition  Probabilities 

Let  n  be  the  number  of  transitions  from  i  to  j  observed  from  N 
ij 

trials  of  a  Markov  chain,  and  let  n.  be  the  number  of  transitions  out 

of  state  i:  n  =  En  . .  Then  the  natural  estimator  for  the  transition 
1  j 

/N  . 

probability  p  is  p  =  n  /n  .  Suppose  two  sets  of  numbers  of  transi- 
ij  ij  ij  i 

tion  data  are  denoted  by  superscripts  1  and  2.  Then  the  intuitive  way 
to  combine  these  data  is  to  simply  add  the  numbers  of  transitions  for 
corresponding  i  and  j 


(1)  ,  (2) 
n  =  n  +  n 
ij  ij  ij 


for  all  i,j 


(13.10) 


Then  the  corresponding  estimators  of  transition  probabilities  p  are 

ij 


(1)  ,  (2) 

n  +  n 

ij _ ij 


(1) 

n 

ij 


ij 


E  & 


D  ,  (2)-| 

+  n.  . 

J  ij  -1 


(1)  ,  (2) 
n  +  n 
i  i 


+ 


(2) 

l 

ij 


(1)  (2) 

n  +  n 
i  i 


(13.11) 
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Some  algebraic  manipulation  puts  this  in  the  form  where  weights  w  and 

w  are  identifiable  as  are  the  estimates  for  the  first  and  second  data 
2 

sets . 


P.  .  = 
ij 


r  (1) 

n 

ij 

(1) 

n 

l  i 


n 


(1) 


(1)  ,  (2) 
n  +  n 
i  i  J 
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(2) 

n 

ij 

(2) 

n 

i 


(2) 


(1)  ,  (2) 
n  +  n 
i  i  J 


(1)  (2) 

n  n 

iJ  ,  ij 

— ; — —  w  +  - —  w 

(1)  1  (2)  2 


(13.12) 


The  question  is  "do  the  weights  w  ,  w^  agree  with  the  a  and  (1  -  a)  de¬ 
termined  earlier?"  Fortunately,  the  answer  is  yes. 

To  see  this,  recall  that  the  formula  for  the  variance  of  the  sample 
success  probability  estimator  in  N  Bernoulli  trials  is  p(l  -  p)/N,  or 
pq/N  when  q  =  1  -  p.  Applying  this  here  gives 


.(1)  .  (1) 
var  p  =  p  q  /n 
ij  ij  ij  i 


(2)  .  (2) 
var  p  =  p  q  /n 
ij  ij  ij  i 


(13. 13) 


The  multiplier  for  the  second  variable  (1  -  a)  is  therefore 


/  (1) 
P.  .q.  ,/n. 

IJ  IJ  1 


(2) 


2  ,  2 

a  +a 

1  2 


r  /  (1)"i  4-  r  /  (2)_i  (1)  4.  (2) 

p  ,q  ,/n  +  p  ,q  /n  n  +  n 

L  ij  ij  l  J  L  ij  ij  i  J  i  i 


(13.14) 


which  agrees  with  w  .  Since  w  +  w  =1,  the  value  of  w  will  agree  also 
therefore,  for  the  special  case  of  combining  transition  information  from 
independent  samples,  the  simple  "add  transitions"  rule  is  optimal  for 
obtaining  estimates  of  p  with  minimum  variance. 

ij 

When  estimates  of  transition  probabilities  are  combined  that  are  not 
determined  from  ratios  n  /n  ,  the  new  estimates  may  not  sum  to  unity. 

ij  i 

For  example,  suppose  transitions  can  be  made  from  state  1  to  states  2  or 
3  in  a  larger  process  that  need  not  be  specified.  Then,  if  two  estimates 
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of  p  are  (0.2,  0.3)  and  of  p  are  (0.8,  0.7) 
12  13  ’ 

ances  (1,2)  for  the  p  estimates  and  (3,4)  for 

J.  £ 

the  weighted  estimates  from  Eq.  (13.5)  are 


with  corresponding  vari- 
the  p  estimates,  then 


12 


1  +  2 


(0.2)  + 


1  +  2 


(0.3) 


0.7 

3 


4  ,  x  3  ,  x  3.2  +  2.1  5.3 

p  =  -  (0.8)  +  -  (0.7)  =  -  =  - 

*13  3+4  3+4  7  7 


The  weighted  estimates  do  not  sum  to  unity: 


20.8, 

P12  +  P13  =  ~  *  1 


A  slightly  revised  optimization  procedure  can  force  the  weighted 

probabilities  to  be  unity.  Let  X  and  X  each  have  mean  p  and  suppose 

1  £ 

2  2 

var  X^  =  CT^,  var  =  a^;  similarly,  let  and  each  have  mean  p  , 

2  2 

with  respective  variances  0 ^  and  a Weighting  factors  a  and  b  are  then 
sought  such  that 


P12  =  .Xj  +  (1  -  a)X2  ,  P13  =  bYx  +  (1  -  b)Y2  , 

p  +  p  =  1  ,  and  (var  p  +  var  p  i  is  a  minimum 

1 J  \  1«J  / 


(13.15) 


Forming  the  Lagrangian,  taking  partial  derivatives  with  respect  to  a  and 

b,  setting  these  derivatives  to  zero,  and  solving  together  with 

p  +  p  =1  results  in 
*12  *13 


a  =  b 


2  2 

a  +a 

2  4 


2  2  2 
0+0+0 
12  3 


2 

+  a 

4 


(13.16) 
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D.  The  Variance  of  the  Probability  of  Success 

Many  questions  arise  when  looking  into  the  variance  of  the  proba¬ 
bility  of  success  p  .  On  the  one  hand,  we  can  assume  that  there  is  some 
probability  of  success  p  — fixed  but  not  known — such  that  success  is  de¬ 
termined  by  tossing  a  coin  with  sides  labeled  success  and  fail  and  which 
comes  up  success  with  probability  p^  and  fail  with  probability  1  -  p^. 

This  Bernoulli  model  yields  a  variance  of  p  (1  -  p  )/N  for  N  independent 

s  s 

trials,  where  a  trial  corresponds  to  a  complete  exercise. 

Other  models,  however,  give  other  variances.  Assume,  for  example, 

some  value  of  success  probability  p  exists  for  each  fixed  set  of  environ- 

s 

mental  parameters,  but  the  environment  varies  randomly.  Then  p^  is  a 

random  variable  with  a  mean  and  variance,  and  its  mean  value  p  is  the 

s 

average  probability  of  success.  Dropping  the  subscript  s  for  simplicity, 

assume  p  is  a  random  variable  with  mean  p  and  variance  v.  The  mean  and 

variance  of  the  estimator  p  =  (number  of  successes)/N  can  be  readily 

worked  out.  Choose  values  of  p  at  random,  independently  from  each 

other.  For  each  p.,  perform  N  trials  (exercises)  each  resulting  in 

i  2 

success  or  failure.  Let  S  =  number  of  successes  using  p  .  Then  S,  the 

i  i 

number  of  successes  in  N  X  N  trials,  is 

1  ^ 

s  =  S1  +  s2  +  ...  +  SN  •  (13.18) 
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Thus,  the  estimate  of  (average)  probability  of  success  is  p  = 
since  there  are  trials  in  all.  The  mean  value  of  p  is 


E(p)  -  E 


Z  v(nin2) 


Ni 


■ixZE(si) 


S/(N  N  ) 

1  2 


(13.19) 


1  2  7=1 


But, 


E(Si)  “  E[E(SilP)]  =  E(v)  =  V<P) 


(13.20) 


Hence, 

N  X  N2E(p) 

E(p)  =  - — -  =  E(p)  =  p  (13.21) 


so  that  p  is  unbiased  as  anticipated. 

Similarly,  by  conditioning  on  p  and  using  the  general  formula 


var  p  =  [E(var  p|p)]  +  var[E(pjp)]  ,  (13.22) 


one  obtains  after  some  algebra 


var  p 


p(l  -  p)  +  var  p 

N  N  N 

12  1 


(13.23) 


Note,  this  is  not  equal  to  the  variance  obtained  when  p  replaces  p  in 
the  Bernoulli  model;  in  fact,  the  Bernoulli  model  estimate  gives  only 
the  first  term  p(l  -  p)/N  N  .  The  actual  variance  is  larger  by  the  non- 
negative  quantity  var  p  X  (N  -  1)/N  N  . 

If  this  model  were  to  be  used  for  exercise  design  purposes,  some 
very  unrealistic  designs  would  result.  Suppose,  for  example,  that  the 
k  exercises  can  be  performed  in  all,  and  that  the  design  problem  consists 
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of  choosing  N  and  N  to  minimize  the  variance  in  Eq.  (13.23).  The 
1  2 

second  term  in  Eq.  (13.23)  is  zero  when  is  chosen  unity;  that  is,  when 
only  a  single  exercise  is  performed  for  each  environment.  The  trouble 
with  the  model  is  that  the  exercise  costs  are  not  considered,  and  other 
benefits  from  the  exercises  are  ignored. 


A  slight  generalization  of  the  above  model  is  easy  to  derive.  Sup¬ 
pose  that  N  separate  environments  can  be  independently  chosen,  where  the 

th 

probability  of  success  is  p  for  the  i  environment.  Perform  N  inde- 

i  i 

pendent  trials  in  environment  i,  and  let  S.  be  the  number  of  successes. 

N  i 

The  total  number  of  trials  is  £  N  ,  which  is  denoted  by  N  .  As  before, 
N  i=l  1  ° 

the  estimate  p  =  £  S  /N  is  an  unbiased  estimate  of  the  average  proba- 

i=l  i  o 

bility  of  success  p.  The  variance  of  p  works  out  to  be 


(13.24) 


Once  again  the  N.  should  all  be  chosen  equal  to  unity  in  order  to 

minimize  the  variance  of  p  because  this  choice  makes  the  second  term  zero 

when  the  sum  of  the  N  is  a  constant.  When  these  N  are  chosen,  the 

i  i 

variance  is  the  p( 1  -  p)/N  ,  which  is  the  Bernoulli  formula  for  variance 

o 

with  p  replacing  p. 

E.  Optimal  Allocation  of  Sampling  in  a  Simple  Markov  Chain 

A  simple  Markov  chain  model  with  four  states  and  constant  (but  un¬ 
known)  transition  probabilities  is  defined  here  to  illustrate  how  exer¬ 
cise  effort  may  be  allocated  in  principle  when  exercise  costs  are  considered 
and  when  minimizing  the  variance  of  the  estimator  for  the  probability  of 

success  p  is  desirable.  The  transition  diagram  with  transition  proba- 
s 

bilities  on  the  appropriate  branches  is  shown  below: 
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1 


START 


Because  there  are  no  loops  in  the  chain,  the  probability  of  success  is 

p  =  ^X2^23  w^en  ^he  Process  always  starts  in  state  1.  The  estimate  of 

p  will  be  the  product  p  p  ,  where  the  carets  denote  estimates  of  p 
s  12  23’  12 

and  pi3. 

Although  the  ASW  operations  being  modeled  always  start  in  state  1, 
we  assume  that  it  is  possible  to  perform  experiments  (exercises)  that 
start  at  either  state  1  or  state  2  and  proceed  for  only  a  single  transi¬ 
tion.  Let  the  cost  of  an  experiment  starting  at  state  1  be  c^,  the  cost 
of  an  experiment  starting  at  state  2  be  c and  the  total  resources 
available  be  c  in  commensurable  units.  Suppose  that  estimates  p  and 
p^3^  are  available  for  the  quantities  p  and  p^.  Variances  of  these 
estimates  are  assumed  to  be  available  also,  with  respective  values  of 

v^  ^  and  v  \  The  experimental  design  problem  is:  Choose  the  number  of 
12  23 

experiments  starting  in  state  1  (n^)  and  number  of  experiments  starting 

in  state  2  (n  )  such  that  the  predicted  variance  of  the  estimate  of  p 
2  s 

is  minimized  subject  to  the  cost  constraint  c  n  +  c  n  =  c. 

(2)  (2) 

The  idea  is  to  obtain  estimates  p  „  and  p  „  from  the  n  +  n  new 

12  *23  1  2 

experiments,  weight  these  estimates  in  the  optimal  linear  manner  as  de¬ 
rived  in  Subsection  B,  and  minimize  the  variance  of  this  weighted  esti¬ 
mator.  Some  assumptions  must  be  made  about  the  variances  this  new  set 
of  transition  probability  estimators  because  they  depend  on  the  unknown 


values  of  the  true  transition  probabilities. 
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If  an  average  value  approach  is  used,  the  predicted  mean  and  vari- 

(2)  (2)  , 

ances  of  p  and  p  (the  estimates  obtained  from  n  trials  starting 
12  23  1 

at  state  1  and  n^  trials  starting  at  state  2)  are: 


(2)  (i)  r.  (in, 

var  p  — P  1  -  p  /n 

P12  12  L  *12  J  1 


(1) 


(D- 


(13.25) 


var 


(2)  (1)  r  (in, 

p  a:  p  1  -  p  /n 

23  23  L  *23  J 


(1) 


(D- 


(13.26) 


(2)-|  (i) 

E  p  —  P 

Lf12  J  *12 


(13.27) 


(2n  (l) 

ELP23  j  *  P23 


(13.28) 


Letting  p  and  p  be  the  predicted  weighted  estimators  for  p  and  p  , 

1 .A  Axj  1^  £%-> 

Eq.  (13.6)  yields  the  predicted  variance 


(1)  (2) 
v  v 
12  12 


var  p 


12  (1)  (2) 

v  „  +  v 

12  12 


+ 


(1)  (1) 

v  p 

12  *12 


[>  -  ^’3 


and  a  predicted  mean 


(2)  (1)  (1)  (2) 
_  _  (D  .  (2)  _  V12  P12  V12  P12 

P12  “  aP12  "  3  P12  (1)  ,  (2) 


v  +  v 
12  12 


(13.29) 


(13.30) 


Similar  expressions  hold  for  the  2  -*  3  transition,  with  subscripts  12 
replaced  by  23  and  n^  replaced  by  n^ 
from,  here  on  since  they  are  all  (1). 


replaced  by  23  and  n^  replaced  by  All  superscripts  may  now  be  dropped 
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The  variance  of  the  product  of  two  independent  random  variables  (X 


and  Y)  is  easily  shown  to  be 

var  (XY)  =  e(x2)e(y2)  -  E2(X)E2(Y)  .  (13.31) 

Applying  this  formula  to  p^  =  p  p  Sives 

var  (ps)  =  e(4)e(4)  -  e2(p12)e2(p23)  .  (13.32) 

Equations  (13.29),  (13.30),  and  (13.32)  imply 


Expression  (13.33)  is  to  be  minimized  with  respect  to  n  ,  n  subject  to 

1.  2 

the  cost  constraint 


°l"l  +  °2n2  ’ 


(13.34) 


and  nonnegativity  constraints 


Vn2  2  ° 


(13.35) 


For  convenience,  define  new  variables  a  .  b  .  a  ,  b  by 
’  12’  12’  23’  23 
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a23  -  P23(1  ‘  P23 ) 
b23  P23 (X  ~  P2s)V23 


Equation  (13.33)  then  takes  on  the  simpler  form  of  Eq.  (13.36) 


var  p  = 


12 


23 


*2  + 


1  h  n  i 

\  23  2) 


+  P 


12 


a 


23  b  +  n 
12  1 


+  P 


23 


12  b  +  n 
23  2 


(13.36) 


The  usual  procedure  for  constrained  optimization  may  then  be  followed. 

Let  \  be  the  Lagrange  multiplier  for  the  cost  constraint  in  Eq.  (13.34). 

Differentiating  the  Lagrangian  var  p  +\(cn  +cn)  partially  with  re- 

s  11  2  2 

spect  to  n^  and  n^,  equating  the  partials  to  zero,  and  doing  some  algebra 
results  in 


T;  I“[ai2(b12+”l)  +  P12(b12+”l)2]  =  [a23(b23+n2)  +  ' 


2  12 


(13.37) 


Replacing  n  by  (c  -  c  n  )/c  on  the  right  hand  side  (RHS)  of  Eq. 
2  112 

(13.37)  and  doing  some  algebraic  manipulation  yields 


RHS  of  Eq.  (13.37)  =  R  +  R  n  +  R  n 

o  11  2  1 


(13.38) 


where 


Ro  =  a23b23  +  a23  t  +  P23(b23  +  f] 

2  \  2 


(13.39) 


C1  2  /  c\Cl 

Rl  =  -a23r-  2P23  b23  +  7T 
2  \  2/2 


(13.40) 


E2  *  P23(C1/C2)‘ 


(13.41) 
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Similarly,  the  left  hand  side  (LHS)  of  Eq.  (13.37)  can  be  expanded  to  give 


LHS  of  Eq.  (13.37) 


L  +  L  n  +  L  n 
o  11  2  1 


(13.42) 


where 


L 

o 


a  c 
23 


2  2 

a  b  +  p  b 
a  c  \  12  12  *12  12y 

12  2 


Co  \ 


(13.43) 


L 

1 


a  c 
23  1 


a  c 
12  2 


+  2p 


2b  ) 

12  12/ 


(13.44) 


L 

2 


a  c 
23  1 


a  c 
12  2 


(13.45) 


Equation  (13.37)  now  becomes 


L 

o 


+ 


2 

+  Vi 


=  R  +  R  n  +  R  n 
o  11  2  1 


(13.46) 


from  which  n^  may  be  found  by  the  quadratic  formula.  In  general,  n^  will 

not  be  an  integer;  but  it  will  be  necessary  to  round  to  an  integer.  If 

n  is  negative,  it  should  be  replaced  by  zero;  if  n  =  (c  -  c  n  )/c  is 
1  2  112 

negative,  the  n^  should  be  replaced  by  zero  and  n^  should  be  recalculated 

from  n  =  (c/c  ) . 

1  2 


Table  13.1  shows  some  results  from  the  optimization  procedure.  Ini¬ 
tial  data  were  calculated  as  though  they  were  from  complete  trials  with 
equivalent  sample  sizes  (ESS) .  The  final  column  shows  the  ratio  of  the 
final  standard  deviation  to  the  initial  standard  deviation.  Since  cost 
ratios  c/c  and  c^/c  were  used  as  parameters,  the  cost  constraint  was 
(c/c)  +  (c/c)  =  1. 
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Table  13.1 


TYPICAL  VALUES  OF  OPTIMAL  N*,N* 
(Not  Rounded) 
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XIV  POSSIBLE  OPERATIONAL  APPLICATIONS 


A.  Effectiveness  Applications 

1.  False  Contacts 

The  nine-state  false  contact  model  developed  in  Section  X  can 
be  used  to  replace  the  traditional,  conditional  probability  model  based 
on  the  sequence  Opportunity  -*  Detect  —  Classify  ->  Attack  -*  Kill  in  a 
false  contact  environment.  When  data  from  complete  exercises  are  used 
to  estimate  parameters,  the  value  of  the  effectiveness  measure  (probability 
of  kill)  will  be  unchanged  but  the  influence  of  false  contacts  will  have 
been  made  explicit.  Among  other  advantages,  this  explicitness  enables 
one  to  do  sensitivity  analysis  by  varying  the  average  time  to  prosecute 
a  false  contact  to  determine  the  impact  on  effectiveness. 

The  27-state  false  contact  model,  which  includes  attacks  on 
false  contacts,  may  also  be  used  to  replace  the  traditional,  conditional 
probability  model.  With  this  model,  weapon  expenditures  on  false  con¬ 
tacts  can  be  estimated  as  well  as  the  probability  of  killing  the  actual 
submarine. 

Many  other  forms  of  false  contact  models  may  be  formulated  by 
using  Markov  methods.  Further,  such  things  as  the  kind  of  the  false 
contact  and  alertment  status  in  the  state  definitions  can  also  be  dis¬ 
tinguished. 

2.  Countermeasure  Environments 

Markov  models  may  be  defined  for  complex  mission  environments 
that  entail  the  use  of  certain  kinds  of  countermeasures.  This  -form  of 
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model  should  be  well-suited  to  model  countermeasure  environments  because 


of  the  crucial  importance  of  information  in  these  environments  and  the 
fact  that  states  of  a  Markov  model  are  readily  definable  in  terms  of  in¬ 
formation.  When  states  have  been  satisfactorily  defined,  countermeasures 
can  influence  events  directly  associated  with  the  basic  model  parameters 
in  two  ways: 


The  countermeasures  influence  the  next  information 
state  in  a  manner  directly  associated  with  the  transi¬ 
tion  probabilities  p  (for  i  ^  j). 

The  mean  time  in  an  information  state  is  directly  asso¬ 
ciated  with  the  holding  time  parameter  p. .. 


An  example  of  this  concept  used  in  analytical  countermeasure 
modeling  is  the  semi-Markov  model  developed  in  Ref.  6.  In  this  model, 
the  important  information  state  was  designated  capture  and  the  mean  time 
in  this  state  (capture  time)  was  the  principal  parameter  used  to  charac¬ 
terize  an  acoustic  decoy. 


The  UPTIDE  series  of  exercises  provides  an  example  of  an  opera¬ 
tional  environment  in  which  states  have  been  tentatively  defined  and  data 
have  been  gathered  on  the  time  spent  in  these  states.  Two  sets  of  states 
are  defined:  one  for  the  Blue  forces  and  the  other  for  an  Orange  sub¬ 
marine.  The  High  Value  Target  (HVT)  information  states  are  defined  in 
terms  of  the  existence  of  false/valid  contacts. 


•  No  false  contacts  and  no  valid  contacts 

•  False  contacts  and  no  valid  contacts 

•  Valid  contacts  and  no  false  contacts 

•  Valid  contacts  and  false  contacts. 


States  for  an  Orange  submarine  were  defined  in  terms  of  both  submarine 
information  and  the  submarine's  activity  relative  to  the  Blue  forces. 
Labels  on  some  of  the  states  were  Lured,  Deceived,  Evading,  Search, 
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Confused,  and  Had  Knowledge  of  the  HVT.  Several  combinations  of  these 
states,  such  as  Lured  and  Deceived,  were  also  used. 

Apparently  little  has  been  done  with  the  collected  data  beyond 
the  tabulation  of  the  percentages  of  time  spent  in  the  various  states  and 
the  number  of  entries  into  each  state.  Because  transition  data  were  not 
tabulated  for  the  exercises,  it  is  difficult  to  determine  how  well  a 
Markov  model  fits  the  data  or  to  perform  a  sensitivity  analysis. 

As  they  are  currently  defined,  these  states  are  probably  un¬ 
suitable  for  effectiveness  modeling.  The  state  space  should  be  more  de¬ 
tailed  to  include  the  traditional  functions  detect,  classify,  and  attack 
and  further  refinements  such  as  a  breakdown  of  detection  into  first  CZ 
detections  and  record  CZ  detections.  Separate  states  should  be  provided 
to  distinguish  these  differing  types  of  detection  opportunities  for  the 
HVT,  other  surface  ships,  and  acoustic  decoys.  Classification  of  surface 
targets  by  submarines  should  also  be  represented  in  the  state  definitions. 
Because  submarines  may  erroneously  fire  on  acoustic  decoys,  the  state 
space  should  reflect  this  potentially  important  contribution  of  decoys 
to  overall  ASW  systems  effectiveness. 

If  a  Markov  model  were  successfully  fitted  to  UPTIDE  data,  some 
important  sensitivity  analyses  could  be  performed  and  quantitative  trade¬ 
offs  could  be  established.  A  sensitivity  analysis  similar  to  that  per¬ 
formed  in  Section  XII  could  give  an  estimate  of  the  number  of  hours  in¬ 
crease  in  average  survival  time  for  each  hour  increase  in  mean  capture 
time.  Examples  of  interesting  trade-offs  are: 

•  The  relationship  between  the  range  at  which  an  acoustic 
decoy  is  detectable  (an  acoustic  power  consideration) 
and  the  length  of  time  that  a  submarine  is  captured 
(a  fidelity  consideration). 
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•  The  relationship  between  the  range  at  which  the  HVT 
can  be  detected  and  the  probability  of  its  being 
misclassif ied. 

•  The  relationship  between  the  probability  of  survival 
of  the  HVT  and  other  quantities  such  as  the  mean  time 
acoustic  decoys  were  operative. 

B.  Other  Applications 

Throughout  this  report,  the  emphasis  has  been  on  ASW  effectiveness 
modeling.  However,  applications  of  the  methodology  to  real  naval  problems 
should  not  be  limited  to  effectiveness  models  because  they  may  also  be 
applicable  to  important  problem  aspects  related  or  ancillary  to  effective¬ 
ness.  Markov  models  are  potentially  valuable  in  the  four  areas  discussed 
below.  These  areas  include  a  dynamic  model  of  the: 

•  Range  between  a  submarine  and  a  HVU 

•  Electromagnetic  radiations  from  an  aircraft  carrier 

•  Submarine  classification  process 

•  Submarine  exposure. 

1.  Distance  Between  a  Submarine  and  a  High  Value  Target 

Because  so  many  tactical  interactions  are  ultimately  a  function 
of  range,  one  quantity  of  special  interest  to  model  dynamically  is  the 
distance  between  two  major  participants  such  as  the  enemy  submarine  and 
an  aircraft  carrier  in  a  given  scenario.  A  dynamic  model  of  this  range 
may  be  useful  in  several  applications;  in  particular,  the  model  may  be  a 
useful  component  in  the  design  of  a  real-time  tactical  decision-making 
aid  for  the  Officer  in  Tactical  Command  (OTC) .  The  model  may  provide 
useful  quantitative  answers  to  the  following  questions.  All  questions 
will  assume  that  some  information  on  the  current  range  to  the  submarine 
is  available  and  expressible  as  a  probability  distribution. 
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•  What  is  the  mean  length  of  time  until  the  submarine 
first  comes  within  two  convergence  zones  of  the  car¬ 
rier?  One  convergence  zone?  Expected  range  of  detec¬ 
tion  via  direct  path? 

•  What  is  the  probability  that  the  submarine  will  not 
come  within  a  specified  distance  (say  30  mi)  of  the 
carrier  in  the  next  24  hour  period? 

•  What  is  the  probability  distribution  of  the  current 
range?  (From  this  distribution,  such  quantities  as 
the  median  current  range,  the  probability  that  the 
range  exceeds  that  at  which  a  UHF  transmission  could 
be  detected  by  the  submarine,  and  the  probability  that 
the  submarine  is  beyond  the  range  of  periscope  detec¬ 
tion  can  be  determined.) 


2.  Radiation  from  an  Aircraft  Carrier 

One  may  consider  the  detectable  electromagnetic  radiation  from 
an  aircraft  carrier  as  a  random  process.  A  model  of  this  process  may  be 
a  useful  component  in  a  larger  model  in  which  the  carrier  is  detectable 
by  any  one  of  several  sensors.  (In  principle,  all  forms  of  detection  can 
be  considered  in  a  single  model  by  relating  sonar  detectability  to  carrier 
speed  and  screw  usage  and  periscope  detectability  to  time  of  day. )  A 
state  of  detectable  radiation  would  be  defined  in  a  Markov  model  in  terms 
of  the  radiating  equipment:  State  1  may  be  occupied  when  there  is  com¬ 
plete  EMCON,  state  2  may  be  occupied  when  HF  radio  is  being  used,  state 
3  may  be  occupied  when  the  surface  search  radar  is  energized,  and  so 
forth.  Obviously,  many  combinations  of  radiators  must  also  be  considered. 

Inherent  in  this  model  are  such  quantities  as  the  mean  total 
time  each  sensor  is  used  during  the  exercise,  the  mean  time  between  suc¬ 
cessive  uses  of  a  given  sensor,  and  the  mean  length  of  each  kind  of 
transmission. 
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3. 


Submarine  Classification  Process 


This  model  would  be  a  dynamic  model  with  absorbing  states  repre¬ 
senting  the  final  classification  used  to  determine  the  parameters  in  a 
confusion  matrix.  (Indeed,  the  absorption  probabilities  found  from  the 
dynamic  model  will  be  closely  related  or  identical  to  entries  in  the 
confusion  matrix.)  Dynamic  quantities  such  as  the  mean  time  to  classify 
a  HVT  correctly,  the  probability  of  ever  holding  an  erroneous  classifica¬ 
tion,  and  the  like  should  be  readily  computable  from  the  model.  States 
will  bear  such  labels  as  "possible  carrier,"  "probable  carrier,"  "certain 
carrier"  in  this  model. 

4.  Submarine  Exposure 

This  model  relates  to  detection  in  a  manner  similar  to  the  air¬ 
craft  carrier  radiation  model.  States  would  be  defined  in  terms  of  de¬ 
tectability  of  the  submarine;  exposure  of  the  attack  periscope  would 
correspond  to  occupying  a  certain  state,  exposure  of  ECM  masts  another 
state,  and  so  forth. 

C.  Recommendations 

(1)  A  research  project  to  apply  the  dynamic  Markov  modeling  methods 
developed  during  this  study  should  be  undertaken  in  an  area  of  current 
interest  to  the  Navy.  The  model  should  be  developed  for  a  class  of  tac¬ 
tical  problems  where  wholly  satisfactory  measures  of  effectiveness  have 
not  yet  been  developed  because  of  the  dynamic  nature  of  the  problem, 
the  complexity  of  the  problem,  or  the  difficulty  of  relating  lower  level 
performance  measures  to  the  overall  effectiveness  measures.  The  Naval 
Analysis  Programs  Office  of  the  ONR  could  sponsor  such  research  as  a 
continuation  of  the  Queueing  Methodology  study. 


250 


(2)  The  technique  of  using  a  Markov  model  to  compare  dissimilar 
models  or  to  assist  in  validating  a  simulation  model  should  be  suggested 
to  Navy  analysts  concerned  with  simulation  validation  and  effectiveness 
modeling. 
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Appendix  A 


RELATED  ASW  MODELING 


1.  General  Discussion 

A  survey  of  recent  analytical  ASW  modeling  that  is  related  to  the 
current  SRI  research  is  presented  in  this  appendix.  The  relationship  be¬ 
tween  these  works  and  the  SRI  research  is  also  discussed. 

Aside  from  the  direct  applicability  to  ASW  situations  as  depicted  in 
this  report,  these  techniques  are  in  some  ways  similar  to  but  in  other 
respects  generalizations  of  applications  of  queueing  and  Markov  or  semi- 
Markov  analysis  appearing  in  other  recent  work  in  the  ASW  field. 

Much  of  the  earlier  work  in  ASW  analysis  has  centered  around  compu¬ 
tation  of  total  system  effectiveness  as  compounded  by  a  host  of  influencing 
factors  that  in  a  sense  may  be  represented  by  a  series  of  necessary  stages 
for  success  of  an  ASW  mission  in  a  logical  sequential  order.  That  is,  an 
approximate  description  of  ASW  functions  is  the  traditional  series  of 
stages  in  so-called  WSE  models: 

Opportunity  -*  Detect  -*  Classify  -*  Localize  -*  Attack  -*  Kill 

As  shown  in  Section  III,  this  series  leads  naturally  to  a  Markov  presenta¬ 
tion  from  the  assumption  that  all  transitions  in  the  series  are  independent 
of  previous  ones  and  additional  exit  transitions  to  a  fail  state  are 
available.  This  traditional  series  of  stages  produces  a  simple  formula 
for  the  probability  of  success: 
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At  this  point,  an  explanation  of  the  gross  effect  of  dynamic  inter¬ 
actions  on  the  nature  of  mathematical  models  as  frequently  used  in  ASW 
analysis  seems  worthwhile.  A  specific  example  is  the  simple  concept  of 
a  multiple  server  queue  used  in  Conolly's  ASW  model.11  The  only  reason 
that  a  queue  has  to  have  a  waiting  line  (in  the  ASW  case,  this  would  con¬ 
sist  of  contacts  not  yet  detected  or  pursued)  is  that  dynamic  interaction 
occurs.  That  is,  if  arrivals  are  distributed  (in  probability)  so  that 
they  can  accumulate  to  more  than  the  "servers"  can  handle  in  some  given 
period  of  time,  then  some  arrivals  would  have  to  wait  before  beginning 
service.  Conolly  assumes  an  infinite  amount  of  service  facilities  (prose¬ 
cuting  destroyers)  and  therefore  experiences  no  waiting  for  service,  but 
he  allows  service  time  to  depend  on  number  of  arrivals  still  in  the  sys¬ 
tem.  More  generally,  crisis  situations  in  ASW  contact  prosecution  may 
entail  excessive  congestion  at  certain  ASW  function  levels  so  that  some 
contacts  might  have  to  wait  to  be  served.  An  accurate  measure  of  threat 
to  forces  protected  by  the  ASW  team  is  essentially  impossible  to  deter¬ 
mine  without  dynamic  analysis  of  the  degree  to  which  unprosecuted  contacts 
can  accumulate,  for  how  long,  at  what  range,  and  so  on.  This  fact  was 
recognized  in  the  MAFTEP  outline  that  led  to  the  current  project  and  has 
been  slowly  permeating  recent  work  on  ASW  effectiveness  modeling.1’11-17 

In  its  early  development,  dynamic  analysis  of  ASW  effectiveness  has 
been  mostly  of  a  qualitative  nature.1 s-1°  A  number  of  new,  advanced, 
analytical  approaches  for  modeling  ASW  effectiveness  have  recently  been 
introduced,  including  decision- theoretic  techniques  (principally  statis¬ 
tical  hypothesis  testing)13’13  and  queueing  theory.11’14’13  To  our  best 
knowledge,  actual  analytical  evaluation  of  ASW  models  before  the  current 
work  has  found  little  application  of  advanced  concepts  beyond  a  Markov 
or  semi-Markov  process  with  a  few  states.16’17  The  analysis  during  the 
first  year's  work  on  the  current  project  entailed  queueing  models  repre¬ 
sented  by  highly  dimensional  vector  Markov  processes  along  with  flow 
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graph  techniques  and  recent  developments  in  successive  overrelaxation 
me thods.1 

The  current  work  centers  mainly  on  discrete- time  Markov  and  semi- 
Markov  modeling  with  moderate  numbers  of  states,  and  the  analysis  indi¬ 
cates  that  considerably  higher  dimensions  could  be  used  without  stepping 
beyond  the  bounds  of  computational  feasibility.  Advanced  techniques  used 
in  the  current  work  include  flow  graph  methods  for  discrete-time  semi- 
Markov  processes  (using  Z- transforms) ,  special  matrix  methods  or  quadratic 
programming  for  least  squares  fitting  of  Markov  process  parameters  to 
observed  data,  and  concrete  examples  of  ASW  models  to  demonstrate  applica¬ 
bility  of  these  techniques. 

The  new  developments  of  the  current  study  relate  to  earlier  ASW 
analyses  in  two  principal  ways: 

•  Interpretation  and  means  of  generalizing  their  approaches  in 
terms  of  more  advanced  techniques. 

•  Motivation  for  and  justification  of  Markov  techniques  applied 
to  ASW  analysis,  as  performed  under  the  present  contract. 

As  expounded  above,  the  general  theory  of  Markov  and  semi-Markov 
processes  leads  to  very  powerful  tools  for  stochastic  process  analysis, 
somewhat  analogous  to  the  technique  of  linearization  in  the  areas  of 
mathematical  programming,  differential  equations,  and  operations  research. 
In  spite  of  much  advanced  formulation  of  new  concepts  in  all  these  areas, 
about  the  only  way  to  obtain  numerical  results  effectively  is  to  Markovize 
a  complex  analytical  formulation.  Again,  a  good  example  of  this  is  the 
successful  computations  in  the  first  year's  work  on  this  contract  with 
Markov  processes  having  a  thousand  states  used  to  analyze  an  ASW  conges¬ 
tion  situation  originally  formulated  in  terms  of  queueing  theory. 
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In  the  sections  below,  the  ASW  analyses  referred  to  above  will  be 
treated  individually  to  explain  in  greater  detail  their  relationship  to 
the  current  work. 

2 .  Wagner  ASW  Model  with  False  Contact  Considerations 

Daniel  H.  Wagner  Associates  recently  concluded  an  investigation  into 
the  possibilities  of  modeling  ASW  performance  with  primary  emphasis  on 
measuring  effects  of  false  contacts.12’13  The  Wagner  ASW  model  subdivides 
ASW  operations  into  the  following  two  categories:  detection  and  classifi¬ 
cation,  and  command  level  decisions  and  actions.  The  interplay  between 
these  two  areas  may  be  viewed  as  a  dynamical  feedback  process,13’13  with 
updated  classification  information  proceeding  to  the  command  and  action 
model  and  sensor  team  instructions  feeding  back  to  the  sensor  model,  as 
indicated  in  Figure  A.l. 


SOURCE  References  12  and  13. 


FIGURE  A.l  BASIC  STRUCTURE  OF  WAGNER  ASW  MODEL 


Within  the  Sensor  Team  block  of  the  flow  diagram  in  Figure  A.l,  the 
Wagner  report  describes  in  greater  detail  how  information  processing 
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interacts  as  the  various  types  of  sensor  teams  and  units  among  one  type 
function  together.  This  is  depicted  in  Figure  A.2.  Basically,  these 


Individual 
Sensor  Teams 


To  CO 


From  CO 


SOURCE:  References  12  and  13. 


FIGURE  A.2  INTERACTION  WITHIN  THE  "INNER  SANCTUM" 
OF  THE  SENSOR  TEAMS 


interactions  are  in  the  form  of  reports  of  contacts  made  or  updatings 
of  characteristics  of  earlier  reported  contacts  for  the  benefit  of  other 
sensor  teams  and  the  supervisors  in  making  or  updating  their  classifica¬ 
tion  decisions.  Thus,  the  arrows  in  Figure  A.2  may  be  interpreted  as 
indicating  flow  of  information  on  a  statistical  basis.  This  viewpoint 
gives  the  first  indication  of  a  relationship  with  the  analysis  of  the 
current  project;  that  is,  the  Wagner  model  may  be  regarded  as  (or  at 
least  approximated  by)  a  Markov  process,  or  perhaps  more  generally  a 
semi-Markov  process,  whose  states  are  described  by  levels  of  information 
content  and  classification  pertaining  to  a  specified  number  of  contacts. 

In  the  first  year's  work  on  this  contract,  four  levels  of  action  on  up 
to  four  contacts  on  actual  submarines  in  the  system  were  considered  in  a 
simplified  dynamical  ASW  model. 

The  Wagner  model  does  not  explicitly  discuss  the  dynamics  of  how 
various  contacts'  penetration  levels  interact  in  the  information  processing 
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system  of  Figures  A. 1  and  A. 2.  Rather,  the  primary  emphasis  is  on  modeling 
the  decision-making  mechanism  by  (Neyman-Pearson  type)  statistical  hypothe¬ 
sis  testing.13’13  The  extensive  formulation  of  state  vectors  and  their 
functional  interrelationship  results  in  a  precise  mathematical  statement 
of  the  relations  among  the  following  quantities  or  logical  (Boolean) 
variables  (for  hypotheses) : 

•  Detection/Classification  Phase 

-  At-sea  environment  factors 

-  Classification  clues  present 

-  Perception  probabilities  for  specific  objects  detected 

-  Hypothesis  that  object  observed  is  of  a  certain  classification 

-  Rejection  probabilities 

•  Command/Action  Phase 

-  Possible  tactics  of  contacts 

-  Hypothesis  that  contact  is  employing  a  certain  tactic 

-  Current  classification  information 

-  Tactical  mission  decisions. 

Many  others  are  omitted  here  for  the  sake  of  brevity. 

For  the  type  of  approach  put  forth  there,  the  Wagner  report  indicates 
an  important  application  to  ASW  simulations.  Presumably,  any  dynamical 
interplay  not  explicitly  considered  in  the  Wagner  model  itself  would  be 
relegated  to  treatment  by  the  time  varying  mechanics  of  the  model  being 
simulated.  Some  dynamical  aspects  of  ASW  modeling  are  introduced  in  the 
Wagner  analysis  in  a  typical  curve  for  temporal  variation  in  perception 
level  (see  Ref.  12,  Chapter  IV),  in  discussions  pertaining  to  implementa¬ 
tion  in  simulations  (see  Ref.  12,  Chapter  V),  and  in  statements  regarding 
assumed  statistics  of  contact  arrivals  and  classification  time  (Ref.  13, 
Appendix  B) . 
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A  possible  useful  extension  of  the  Wagner  work  might  be  an  attempt 
to  carry  oveV  the  analysis  to  a  dynamic  level  by  suitable  embedding  of 
Markov  chains  as  was  done  on  this  contract  in  another  context.  An  out¬ 
line  of  how  this  might  be  achieved  follows  below. 

The  first  step  necessary  for  Markov  representation  of  ASW  operations 
as  described  in  the  Wagner  reports  is  to  decide  on  what  ASW  events  may  be 
set  apart — or  perhaps  combined  with  others — to  form  distinct  states  of 
the  system.12’13  In  doing  this,  it  is  also  necessary  to  consider  whether 
the  system  can  be  logically  regarded  as  transiting  from  one  of  these 
states  to  another  without  too  much  interference  from  still  other  omitted 
factors.  These  could  be,  for  example,  long-term  value  information  about 
a  contact  determined  early  in  the  detection  to  prosecution  sequence. 

The  Wagner  model  was  already  formulated  in  terms  of  a  Markov  process 
in  the  simulation  of  Ref.  13— Appendix  B.  In  fact,  16  states  were  clearly 
delineated,  but  only  simulation  was  attempted.  Our  model  was  very  closely 
related  in  that  progress  of  contacts  "through  the  system"  described  the 
state  of  the  system,  and  transitions  between  states  were  similarly 
described  although  we  did  not  consider  as  many  states  and  transitions  for 
a  given  contact.  The  much  higher  dimensionality  in  our  model  arose  from 
combining  possible  states  of  various  simultaneous  contacts.  Hence  this 
feature  appears  to  be  the  next  step  in  generalizing  the  Wagner  model 
toward  a  more  realistic  one,  and  the  efficient  computational  techniques 
used  in  our  work — namely,  flow  graph  methods,  Successive  Over  Relaxation 
methods,  or  direct  Laplace  transform  evaluation  and  inversion — would 
succeed  in  arriving  at  solutions  for  an  extended  model  along  the  lines 
of  the  basic  Wagner  approach. 


261 


3. 


Operations  Research  Incorporated  ASW  Queueing  Model 


A  parallel  study  to  the  current  contract  was  aimed  at  modeling  ASW 
operations  of  a  patrol  submarine  in  terms  of  queueing  concepts  and  tech¬ 
niques.  At  its  outset,  a  queueing  structure  was  formulated  similar  to 
our  unit  model  of  the  first  year's  work  on  this  contract.14  Later, 
emphasis  changed  to  modeling  of  the  classification  process.15’ 16  An 
analytical  formulation  for  the  probability  of  classifying  by  a  given  time 
was  also  worked  out,  taking  multiple  contacts  into  consideration.  Some 
generally  useful  concepts  for  ASW  modeling  introduced  by  ORI  in  their 
model  were  the  "classification  window"  and  the  "growth  of  information" 
idea  for  controlling  the  incidence  of  actual  ability  to  classify.  These 
concepts,  although  not  formally  stated,  appear  to  be  the  basis  of  other 
ASW  analyses,  as  will  be  indicated  further  below. 

4.  Semi-Markov  Mission  Analysis 

Semi-Markov  models  of  aircraft  operations  in  ASW  and  general  target 
search,16’ 17  were  recently  formulated  with  several  states  to  represent 
various  reasonably  distinguishable  stages  of  such  operations.  Laplace 
transforms  were  used  in  both  cases  in  ways  similar  to  the  applications 
on  a  single  unit  ASW  model.1  Flow  graphs  were  used  to  aid  formulation 
and  organize  computations,  as  in  our  work,  and  simple  examples  were  worked 
out.  Solutions  were  obtained  analytically  in  closed  form,  by  inversion 
of  some  elementary  Laplace  transforms. 

Both  these  models  involve  stages  of  a  process  (search  for  targets) 
that  are  passed  through  more  or  less  independently  of  activity  at  other 
stages,  thereby  leading  to  a  fairly  good  Markovian  model.  The  situation 
in  Ref.  16  is  simplified  further  than  in  Ref.  17  by  the  assumption  that 
only  one  contact  held  at  any  one  time.  The  possibility  of  false  contacts 
is  handled  by  considering  states  of  "disturbance,"  which  can  be  interpreted 
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as  either  diversion  to  pursuing  an  attacker  on  the  target  searching  unit 
or  changing  over  to  a  possible  target  of  perhaps  greater  importance  to 
the  search  mission.17 

5 .  Semi-Markov  Models  of  Search  in  the  Presence  of  Decoys 

J.  M.  Moore  of  SRI  has  recently  introduced  a  three-state  and  four-  ' 
state  semi-Markov  model  for  a  patrolling  submarine  versus  HVTs  and  decoys 
operating  within  a  specified  area.6  As  in  semi-Markov  mission  analysis, 
states  were  selected  to  represent  well-distinguishable  operational  con¬ 
figurations  with  transitions  between  them  being  clearly  defined  and 
justifiably  approximated  by  Markov-type  behavior.  Although  Moore's  model 
has  a  small  number  of  states,  it  is  firmly  founded  on  an  operational 
basis,  with  consideration  of  a  reasonably  detailed  structure  of  searcher 
and  evader  maneuvers  to  arrive  at  formulas  for  parameters  in  the  semi- 
Markov  model  in  terms  of  measurable  or  at  least  known  or  postulated  opera¬ 
tional  data.  In  this  sense,  his  model  is  more  practical  than  our  unit  ASW 
model  of  the  previous  year's  work  under  the  current  contract;  our  model's 
semi-Markov  structure  was  far  richer  (in  fact,  up  to  a  thousand  states 
were  allowed),  but  this  structure  was  arrived  at  by  analysis  of  a  postu¬ 
lated  operational  description  that  was  quite  elementary.  Our  actual 
operational  states  were  few  in  number  (describing  the  number  of  contacts 
held),  and  the  analytical  model's  states  arose  from  strongly  distinguish¬ 
able  combinations  of  where  individual  contacts  are  located  in  the  model's 
partitioning  of  events  into  stages  (detection,  localization,  and  such). 

Another  important  feature  of  Moore's  work  is  that  a  parallel  simula¬ 
tion  study  was  conducted  at  SRI10  for  comparison  of  model  accuracy  and 
validation.  This  simulation  model  is  discussed  and  used  in  Section  XII 
of  this  report. 
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As  with  the  other  related  analyses,  a  significance  of  our  develop¬ 
ments  using  flow  graphs  and  other  multidimensional-oriented  techniques 
is  a  possible  approach  to  computations  when  Moore's  type  of  model  is  ex¬ 
panded  to  greatly  larger  number  of  states.  From  the  other  direction, 
Moore's  reduction  of  an  operational  situation  to  a  low-dimensional  model 
is  a  valuable  technique  to  maintain  closer  ties  with  "the  real  world"  in 
modeling.  A  similar  approach  was  begun  but  not  completed  in  our  single 
unit  ASW  model  when  it  was  found  that  too  many  states  arose  in  a  Markov 
model  as  a  result  of  combinations  of  possible  contact  locations. 

6 .  Passive  Sonar  Classification  Model 

A  mathematical  model  of  contact  classification  by  passive  means  was 
developed  at  SRI  and  used  on  several  projects  in  the  ASW  area.18  The 
basis  of  this  model  is  consideration  of  a  sequence  of  information-gathering 
stages  related  to  acoustic  and  kinematic  aspects  of  passive  sonar  search. 

As  quality  and  quantity  of  information  fluctuates  stochastically,  a  ten¬ 
dency  toward  some  postulated  and  recognizable  pattern  class  may  emerge 
and  be  interpreted  as  a  cue  to  update  current  classification  of  a  possible 
contact. 

This  basis  is  similar  to  the  ORI  approach  previously  referred  to  and 
also  to  our  expanded  ASW  unit  model  (Section  II-C).  The  key  concept  in 
all  three  cases  is  that  of  growth  of  information  as  a  measure  to  be  related 
through  a  dynamical  model  to  a  decision  process.  In  our  work  and  the 
parallel  ORI  efforts,  the  dynamical  aspects  were  put  in  the  form  of  a 
Markov  process.  There  is  ground  to  believe  that  the  SRI  passive  sonar 
classification  model  could  be  represented  in  Markov  or  semi-Markov  form 
for  application  in  mission  analyses  along  the  lines  of  Refs.  6,  12-17, 
thereby  bringing  into  such  models  a  more  realistic  representation  of 
acoustic  signal  processing.  A  good  approximation  to  the  complex  process 
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described  in  Ref.  18  might  be  to  consider  "lumped  states"  created  by 
conditions  of  acoustic  activity  leading  to  the  "pattern  classes"  considered 
there,  and  to  regard  changes  in  recognized  patterns  as  Markov-type  transi¬ 
tions,  provided  that  these  could  be  shown  to  exhibit  suitable  properties, 
especially  independence  of  past  history  of  pattern  transitions. 

7 .  Conolly  Model  of  ASW  in  Terms  of  Congestion  Theory 

An  elementary  queueing- type  model  of  ASW  operations  was  proposed  by 
Conolly  for  the  situation  of  an  ASW  force  attempting  to  clear  a  pro¬ 
tected  area  of  enemy  submarines.11  The  model  assumed  Poisson  arrivals, 
i.e.,  detections  of  submarines  in  general  and  independent  service  time 
distributions  (for  the  contact  prosecutions).  A  method  for  calculating 
the  probability  distribution  of  time  to  clear  the  protected  area  was  de¬ 
rived  in  closed  analytical  form.  In  none  of  our  more  complex  models  were 
analytical,  closed-form  solutions  ever  found  to  be  practical.  Numerical 
solutions,  however,  were  always  feasible. 

8.  SHAREM  Analysis — Markov  Models  of  an  ASW  Screening  Mission 

A  nine-state  Markov  model  of  a  typical  ASW  screening  scenario  was 
used  in  post-analysis  of  a  SHAREM  exercise  sequence. 19' 20  Standard 
Markov  absorbing  state  probability  calculations  were  used  to  analytically 
formulate  MOEs  and  to  evaluate  sensitivities  with  respect  to  transition 
probabilities.  Exponential  distributions  assumed  for  contact  holding 
times  were  found  to  be  quite  consistent  with  exercise  observations. 20 
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SELECTION  OF  PARAMETER  oj 


The  range  derived  for  u)  was  thought  to  lie  in  the  range  1  <  u)  <  2, 
when  the  eigenvalues  of  the  iteration  matrix  are  all  real.31  However, 
it  was  later  realized  that  the  optimal  parameter  u)  may  sometimes  have  to 
be  less  than  1  when  complex  eigenvalues  exist  for  the  iteration  matrix 
labeled  Q  in  Appendix  B-2  of  Ref.  1.  Since  investigation  subsequent  to 
initial  work  revealed  the  presence  of  complex  eigenvalues,  a  value  of 
U)  =  0.9  was  tried,  and  the  number  of  over-relaxation  iterations  per  inte¬ 
gration  step  was  reduced  from  about  13  to  about  6.  Some  effort  was  spent 
on  analyzing  the  relationship  between  uu  and  the  convergence  rate,  and 
Figure  B.l  demonstrates  the  loci  of  eigenvalues  of  the  iteration  matrix 

£  (defined  further  below)  as  U)  changes,  assuming  an  approximate  model 
U) 

of  the  convergence  process.  That  is,  we  considered  a  reduced  form  of  the 
actual  matrix  of  the  almost  5000  equations  to  be  solved,  namely: 


I  AAt 


-I  I 


by  interpreting  the  scalar  number  a  as  representing  the  largest  (negative) 
eigenvalue  of  the  Markov  process  matrix  A,  multiplied  by  the  time  step  /\t, 
and  writing 
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1  a 


a 

6 

-a 

6 

-1  1 


FIGURE  B.1  ROOT  LOCI  FOR  VARIOUS  W  VALUES 
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This  corresponds  to  taking  four  terms  in  the  continued  fraction  expansion 

O' 

of  e  and  is  associated  with  five  nodes  of  the  flow  graph 


a/10 


a 

representing  y  =  e  x.  (We  tried  using  nine  terms  of  this  flow  graph  for 
better  accuracy,  but  no  improvement  resulted  in  overall  computational 
effort. ) 

Analyzing  the  five  eigenvalues  of  the  over-relaxation  iteration 

matrix  Z  and  using  the  above  matrix  Q/ ,  where 
0) 

Z  =  [u)F  -  (1  -  U))D]  (D  +  o>E)-1 
uo 

[see  Ref.  1,  Eq.  (B-8a)]  gives  an  equation  for  four  of  these  eigenvalues 
(one  of  them  is  (jo  -  1)  as  follows: 

2  2  2 

(\  +  uo  -  1)  =  n  , 


where  p  is  an  eigenvalue  of  the  upper  and  lower  triangular  parts  E  +  F 
of  the  matrix  Q  in  Ref.  1.  Hence,  after  some  manipulation,  it  can  be 

shown  that  the  other  four  eigenvalues  of  Z  (being  a  5  X  5  matrix)  are 

UD 

given  by  the  solution  to  the  polynomial 


(  2  1 

2  2  4  ) 

4|]J  +  4 

3 

\  4*  * 

6(uj  -  1)  -  7-  u)  -  -  1)> 

2 

12  I 

I  O'  2  )  2  4 

-  j  —  a)  -  4 a)  4-  4  j ( uo  -  1)  X  +  ((JO  -  1)  =  0 
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When  to  =  1,  we  have  the  reduced  situation 


4 

X 


2 

+  —  \2 
12 


0 


which  forces  two  of  the  eigenvalues  to  be  0.  This  explains  why  the 
straightforward  "block  Jacobi"  method  (without  any  relaxation  factor, 
either  <1  or  >1)  is  not  too  good  for  our  problem  because  it  implies  de¬ 
generacy  of  eigenvalues.  The  optimal  u)  occurs  when  the  coefficients  of 

o 

3 

\  and  X  vanish,  for  then  the  equation  becomes 
34  ,  _2  8(u)  -  l)2 

T  ° - i — 

00 

2  , 

where  U)  is  found  in  terms  of  ey  by  ckd  /8  =  u)  -  1,  i.e., 


(we  take  the  minus  sign  to  get  our  U)  >  0) .  For  u)  =  u)  ,  all  eigenvalues 

o 

are  of  equal,  absolute  value  because  they  evidently  come  in  squares  of 

conjugate  pairs;  thus  clearly,  without  having  to  solve  the  above  equation, 

4 

the  product  of  the  magnitudes  is  known  to  be  the  term  (u>  -  1)  .  Hence, 
the  optimal  convergence  rate  is  j oo  —  l|. 

Now,  the  time  steps  were  taken  to  be  At  =  0.01,  and  the  highest  terms 
appearing  in  the  large  matrix  A  are  -243  on  the  main  diagonal.  Hence,  the 
largest  eigenvalue  (in  magnitude)  of  the  matrix  AAt  can  be  estimated  at 
-2.  When  this  value  is  used,  the  optimal  computed  u)q  is  0.83.  Amazingly, 
the  best  convergence  obtained  experimentally  by  the  integration  program 
was  for  uj  2:  0.9,  with  about  5  iterations  per  step,  while  u)  =  1.1  took  13 
steps.  Thus,  the  analytical  evaluation  of  the  best  u)  corroborated  the 
experimental  results. 


2  4 

X  +  (tu  -  1)  =0 


4 

X  + 
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MASON'S  RULE 


A  convenient  method  is  available  for  determining  transmissions  (or 
gains)  between  any  pair  of  nodes  in  small  flow  graphs  by  inspection.  In 
its  algebraic  form,  the  formula  resembles  Cramer's  rule  for  solving  simul¬ 
taneous  linear  equations. 

Each  flow  graph  has  a  graph  determinant  the  value  of  which  remains 
invariant  under  various  rules  for  manipulating  flow  graphs  to  obtain 
equivalent  flow  graphs.  To  evaluate  the  graph  determinant,  it  is  neces¬ 
sary  to  find  the  path  transmission  of  every  loop  in  the  flow  graph,  where 

a  loop  is  a  sequence  of  (directed)  branches  from  a  node  back  to  itself 

* 

without  repeating  any  other  node.  Figure  C. 1  has  three  loops: 

•  Node  1  to  node  1,  with  loop  transmission  p^. 

•  Node  1  to  node  2,  then  back  from  node  2  to  node  1,  with  loop 

transmission  p,  p  , . 

12  21 

•  Node  1  to  node  2  to  node  4  to  node  3  back  to  node  1,  with 

loop  transmission  p. „p„  .p  .„p„, . 

H  *12*24^43*31 

The  loop  transmission  is  denoted  by  L  .  L  ,  L  ,  ...,  L  in  the  general 

12  3  n 

case;  the  graph  determinant  is  then  obtained  by  expanding  the  product 

(1-Li)(1-L2)'"(1-L„) 

and  deleting  all  terms  that  contain  products  of  touching  loops.  In  the 

example  of  Figure  E.l,  of  the  three  loops  L  ,  L  ,  L  ,  any  selected  pair 

1  Z  u 


A  path  transmission  is  the  product  of  the  transmissions  of  the  branches 
comprising  the  path. 
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I 


24 


FIGURE  C.1  TRANSITION  DIAGRAM  FOR  A  TYPICAL  MARKOV  CHAIN 


touch;  therefore, 


A  1  L1  ~  L2  ~  L3  1  P11  P12P21  ”  P12P24P43P31 


An  equivalent  formula  for  A  is 


A  =  1  - 


^T'  L  L 

Lu  j  k 


Ev 


L  L  +  .  .  . 

i  J  k 

all  loops  k  all  pairs  of  nontouching  all  triplets  of 

loops  j  and  k  nontouching  loops  i,j,k 


Mason's  rule  for  calculating  transmissions  between  selected  nodes  i 
and  j  in  an  arbitrary  flow  graph  is  given  by  the  formula 

K 


k=l 


Vk 


gij 


where  there  are  K  open  paths  from  node  i  to  node  j;  t  is  the  path  trans- 

K 

th 

mission  of  the  k  path,  A  is  the  graph  determinant,  and  A  is  the  graph 

K 

th 

determinant  of  the  subgraph  corresponding  to  path  k.  The  k  subgraph 
is  obtained  by  eliminating  the  open  path  k  and  all  loops  touching  this 
path;  if  all  loops  are  killed  by  an  open  path,  the  determinant  of  its 
subgraph  reduces  to  unity. 
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As  an  example  in  Figure  A.l,  the  transmission  from  node  1  to  node  2 
can  be  determined  to  be  p  /A  because  there  is  a  single  path  of  trans¬ 
mission  p  from  node  1  to  node  2  and  that  path  kills  all  the  loops  so 
that  the  determinant  of  its  subgraph  is  unity.  The  transmission  from 
node  2  to  node  1  is  more  complex.  There  are  two  open  paths  with  trans¬ 
missions  p  and  ppp  ,  and  since  each  of  these  open  paths  kills  all 

ZX  Zft  tl  O  o  1 

the  loops  in  the  graph, 


(a  XI 

1  +  |  a  a  a  ^ 

x  1 

l  21 

/  \  24  43  31/ 

g21  A 


a  +  a  a  a 
21  24  43  31 


a  -aa  -  a  a  ,  a  a 
11  12  21  12  24  43  31 
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LEAST  SQUARES  AND  QUADRATIC  PROGRAMMING 

1.  Least  Squares  Programming 

The  classical  least-squares  problem  can  be  formulated  as  follows: 
Assume  the  linear  model 


y  =  Hx  +  z 


(D.l) 


where 

x  is  an  unknown  constant  vector  (n  X  l) 
y  is  an  (m  X  1)  observable  random  vector 
H  is  an  m  X  n  matrix  of  full  rank 
z  is  an  m  X  1  error  vector  such  that  E(z)  =0. 

Then  the  sum  of  the  squared  deviations  (errors)  is 

zTz  =  (y  -  Hx)T(y  -  Hx)  .  (D.2) 


Differentiating  with  respect  to  x,  equating  the  derivative  to  zero,  and 
solving  for  x  as  in  Section  VI  gives  the  solution 


(D.3) 


A  generalization  from  Aitken  assumes  that  the  covariance  matrix  of 
the  errors  is  known 
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E^zzT)  =  V 


(D.  4) 


-1 

Then  the  scalar  to  minimize  includes  V  ;  specifically  minimize 


(y 


T  -1 

Hx)  V  (y  -  Hx) 


(D.  5) 


The  solution  to  this  generalized  least  squares  problem  is 


/  T  -1  \-1  T  -1 
x  =  IH  V  H  J  HV  y 


(D.  6) 


This  formula  may  also  be  used  to  weight  the  error  components  by  choosing 
V  to  be  a  suitable  diagonal  matrix  with  positive  components. 

It  is  sometimes  convenient  to  place  additional  linear  constraints  on 
the  unknown  variables  x  in  the  model  consisting  of  Eqs.  (D.l)  and  (D.4). 
Assume  these  constraints  are  expressible  as 


Ax  =  k 


(D.7) 


where  A  is  a  known  constant  matrix  of  dimensions  p  x  n,  k  is 
stant  vector  of  dimensions  p  x  1,  and  rank  A  =  p  <  n.  Then 
to  the  constrained  least  squares  model  in  Eqs.  (D.l),  (D.4), 

-i-l 

-1 

/  T  -1  \  T 

A  H  V  H)  A  (k  -  Ax) 


a  known  con- 
the  solution 
and  (D.7)  is 

(D.  8) 


where  x  is  the  solution  to  the  unconstrained  problem  given  in  Eq.  (D.6). 
If  the  constraints  are  not  binding,  the  quantity  multiplying  (k  -  Ax)  in 
Eq.  (D.8)  will  be  zero. 
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The  covariance  matrix  of  x  is 


(D.9) 


where 

- 1  i-  - 1  ~i 

/  T  -1  \  T  /  T  -1  \  T 

B  =  I  -  (H  V  H J  A  A(H  V  A  A  .  (D.10) 

The  constrained  least  squares  formula  given  here  may  be  found  in 
Ref.  22. 


2.  Quadratic  Programming 

Least  squares  methods  given  above  do  not  incorporate  linear  inequality 
constraints;  in  particular,  the  components  of  the  vector  x  cannot  be  con¬ 
strained  to  be  nonnegative.  Quadratic  programming  may  be  regarded  as  an 
extension  of  constrained  least  squares  in  which  nonnegativity  constraints 
can  be  introduced.  For  purposes  here,  a  quadratic  program  can  be  formu¬ 
lated  as  a  minimization  problem: 

T  T 

minimize  z  =  x  Dx  +  c  x  (D. 11) 

subject  to  constraints 

Ax  =  k  and  x  a  0  (D.12) 

where  c  and  k  are  known  constant  vectors  and  D  is  a  symmetric  and  positive 
semidefinite  known  matrix  of  constants.  (In  least  squares  applications, 
these  conditions  on  D  are  automatically  satisfied.) 
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When  it  is  desired  to  minimize 


d2  =  (y  -  Hx)T(y  -  Hx)  ,  (D.13) 

the  right  hand  side  is  first  expanded  to  obtain 

2  T  T  T  T  , 

d  =  x  H  Hx  -  2y  Hx  +  y  y  .  (D.14) 

T 

The  matrix  D  is  identifiable  as  H  H,  and  the  vector  c  is  identifiable  as 

T  T 

-2y  H.  The  constant  term  y  y  may  be  neglected. 

Quadratic  programming  problems  may  be  solved  by  modifying  a  simplex 
code  used  for  solving  ordinary  linear  programs.  The  modifications,  al¬ 
though  few  in  number,  are  not  easy  to  implement.  A  quadratic  programming 
algorithm  suitable  for  small  problems  was  developed  during  the  research; 
regrettably,  it  could  not  be  applied  to  problems  of  realistic  size. 

The  computational  effort  for  linear  programs  is  known  to  be  roughly 
3 

proportional  to  m  ,  where  m  is  the  number  of  rows.  A  quadratic  program 
with  m  constraints  and  n  variables  requires  about  the  same  amount  of 
computational  effort  as  a  linear  program  with  (m  +  n)  rows.  Typical 
linear  programming  applications  have  about  three  times  as  many  columns 
as  rows,  therefore  (m  +  n)  is  about  4  m.  Cubing  this  gives  the  approxi¬ 
mation 


(running  time  of  a  quadratic  program)  sj  k64  m  , 

where  m  is  the  number  of  constraints  and  k  is  a  constant.  This  relation¬ 
ship  should  be  considered  carefully  in  determining  the  number  of  variables 
and  constraints  to  use  in  quadratic  programming  applications. 
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THEOREMS  ON  PARAMETER  ESTIMATION 

In  Section  VII,  a  set  of  theorems  concerning  some  basic  identities 
arising  in  the  estimation  of  parameters  of  a  Markov  chain  were  cited  and 
applied.  In  this  appendix  the  theorems  are  precisely  stated,  proved,  and 
illustrated  with  a  numerical  example. 

Suppose  an  arbitrary  random  process  has  N  states,  where  states  1,  2, 
...,  M  are  transient  and  states  M4-1,  M+2,  ...,N  are  absorbing.  Let 
data  from  this  process  be  given  for  a  set  of  n  complete  trials,  where 
complete  means  each  of  the  n  trials  begins  in  state  1  and  continues 
until  absorption  occurs  in  some  absorbing  state.  A  path  corresponding 
to  a  complete  trial  is  therefore  an  ordered  sequence  of  integers  of  the 
form 


1  t  t  t  . . .  t  a 
12  3  n 


) 


where  each  t  is  an  integer  from  1  to  M  and  a  is  an  integer  in  the  range 

j 

M  +  1  5  a  ^  N.  The  path  represents  the  sequence  of  states  entered  when 
read  from  left  to  right — since  "a"  represents  absorption  in  state  a  the 
path  is  completely  described. 

The  random  process  is  arbitrary.  Nevertheless,  we  will  apply  a 
Markov  chain  model  with  the  same  states. 

Following  Markov  chain  estimation  theory,  we  suppose  transition 
probabilities  p. .  are  estimated  from 
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if  n  >  0 
i 


if  n  =  0 
i 


(E.l) 


where 


n  =  observed  number  of  transitions  from  i  to  j 
ij 

N 

n  =  S  n  =  number  of  transitions  out  of  state  i. 

1  j=l 

The  n  are  readily  computed  from  the  path  data  by  counting  transitions. 

ij 

Now  we  partition  P,  the  matrix  of  estimated  transition  probabilities,  as 
in  Section  V. 


/N  /S 


(E.2) 


A  A 

where  Q  is  M  X  M,  R  is  M  X  (N  -  M) ,  and  I  is  the  identity  matrix  of  dimen¬ 
sion  (N  -  M)  X  (N  -  M). 


According  to  Markov  theory  (Section  V-D) ,  the  true  absorption  proba¬ 
bilities  a  are  found  by  calculating  A  =  (a  ) 
ik  ik 


A 


(I  -  Q)-1R 


(E.3) 


when  Q  and  R  are  the  true  transition  probabilities. 


1 .  Theorem  1 

When  Q  and  R  in  Eq.  (E.3)  are  replaced  by  their  estimates  from  Eq. 

A 

(E.2),  then  the  first  row  of  the  estimator  A, 

A  =  (I  -  Q)-1ft  ,  (E. 4) 
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is  identical  to  the  row  vector  of  ratios 


S 

M+2 


) 


where  S  .  =  number  of  times  (out  of  n^  trials)  that  the  process  is  ob¬ 
served  to  be  absorbed  in  state  j  (j  =  M  +  1,  N).  In  other  words, 

the  Markov  estimates  of  absorption  probabilities  are  identical  to  the 
ratios  (number  of  times  process  is  absorbed  in  j/number  of  trials),  which 
are  computable  directly  from  the  given  data  by  ignoring  all  internal 
transitions  and  simply  counting  numbers  of  entries  into  the  respective 
absorbing  states. 


a.  Proof 


Let  state  0  be  an  artificial  state  from  which  transitions  to 
state  1  (the  starting  state)  are  always  made.  Using  the  n  as  defined 

ij 

above,  a  "balance  equation"  Eq.  (E.5)  can  be  written  at  each  node 


M 

n  +  >  '  n 
01  Z—f  kl 
k-1 


M 


ki 


k=l 


(at  state  1) 


(at  state  i  =  2,3,..,,M) 


(E.5) 


These  relations  merely  state  that  every  sample  path  entering  a  transient 
state  also  leaves  it.  By  summing  the  number  of  entries  into  an  absorbing 
state  over  absorbing  states,  we  obtain  Eq.  (E.6) 


(E.6) 


This  relation  says  that  all  paths  terminate  in  some  absorbing  state. 
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Define 


x 


0 


01 


(E.7) 


N 


■i-2X 


x  =  7  n  =  number  of  transitions  out  of  (or  into)  state  i 

Lj  - 

j=l 

i  =  1,2, ...  ,M  (E.8) 


M 

x  n  = 

i  ki 


k=l 


number  of  transitions  into  state  i 


i  =  M  +  1,M  +  2,  .  .  .  ,N 


(E.  9) 


Then  rewrite  Eq.  (E.5)  to  obtain  Eq.  (E. 10). 


M 


n 


ki 


x_  +  J  -  X  =  X 

0  x  k  1 

k=l  k 


M 


at  states  i  =  2,3,... ,U 


(E. 10) 


j=l  \  J 

Equation  (E.6)  becomes 


IV 

E 

j=M+l 


x  =  n  =  x 
j  01  0 


(E. 11) 


Let 


f 

ji 


i=l 


for  i  =  1,2,.. ,,N;  j  =  1,2,...,M 


Then  Eqs.  (E.10)  and  (E.ll)  become  Eq.  (E.13) 


(E. 12) 
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M 


<  =  X  +  f  X 

1  0  ji  j 


j=l 


M 


x  =  y*  f  x 
i  Z—/  Ji  j 


( i  =  2,3, ... ,M) 


J=1 


N 


X  = 
0 


x.  from  Eq.  (E.ll) 


(E. 13) 


j=M+l 


and  from  the  definitions  of  x.  in  Eq.  (E.9)  for  absorbing  states 


M 


M 


x .  =  ^  — —  x  =  ^  f  x  (i=M+l,M+2,...,N) 

1  ZmU  x  k  ZmJ  ki  k  ' 

k=l  k  k-1 

Combining  Eqs.  (E.13)  and  (E.14)  gives  Eqs.  (E.15)  and  (E.16) 

M 


(E.14) 


x  =  x  +  V'  f  x. 
1  0  Z-J  Jl  J 


(E.15) 


j=l 


M 


x.  =/  f..x.  (i  =  2,3, . . . ,M, . . . ,N) 

l  ji  j 

3=1 


(E.16) 


But  Eqs.  (E.15)  and  (F.16)  are  precisely  the  equations  that  are 
solved  using  the  flow-graph  approach  to  absorption  probabilities  when  the 
true  transition  probabilities  p  are  replaced  by  the  f  (Ref.  5,  pp.  161- 

ij  ij 

162,  200).  Therefore,  the  solution  of  the  flow-graph  equations,  which  are 
transmissions  in  the  flow  graph,  are  the  ratios  x^/x^  when  all  other  vari¬ 
ables  are  eliminated.  Choosing  one  absorption  state  (say  N)  as  the  Success 

state,  the  ratio  x  /x  is  also  the  direct  estimate  given  in  Eq.  (E.17). 

NO 

M 


x  l  md  iN 

N  i=l  number  of  paths  terminating  in  state  N 

x  n 

0  01 


total  number  of  paths 


(E.17) 
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Therefore,  the  estimate  of  success  probability  obtained  when 
using  the  flow-graph  approach  is  identical  to  the  simpler  estimate  on  the 
right  side  of  Eq,  (E.17).  Since  the  flow-graph  approach  can  be  easily 
shown  to  give  the  same  result  as  the  matrix  inversion  method  of  Eq.  (E.4), 
the  proof  is  complete. 


b.  Example  of  Theorem  1 

In  a  four  state  Markov  chain  model,  let  states  3  and  4  be  ab¬ 
sorbing  and  states  1  and  2  be  transient.  Suppose  the  given  set  of  com¬ 
plete  trials  (paths)  are  given  by 

11123 

12224 

12123 

1213 

14 


These  path  data  are  generated  by  an  arbitrary  process,  not  necessarily 
Markov. 


The  numbers  of  transitions  implied  by  these  paths  are  shown  on 
the  transition  diagram  of  Figure  E.l.  The  estimated  transition  matrix 
P  therefore  is: 
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I  -  Q  and  (I  -  Q)  1 


are  readily  calculated 


Having  made  these  Markov  calculations,  we  observe  that  the  (1,1)  element 

A 

of  A  is  identical  to  the  ratio 


number  of  paths  terminating  in  state  3  3 

number  of  paths  ~  5 

and  the  (1,2)  element  is  identical  to  the  ratio  2/5  as  guaranteed  by 
Theorem  1. 

-N  -1 

The  elements  of  (I  -  Q)  calculated  in  Example  1  can  be  used 
to  illustrate  Theorem  2  which  is  stated  below.  Recall  from  Markov  theory 
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that  the  (i,j)  element  of  (I  -  Q)  is  the  mean  number  of  entries  into 
state  j  before  absorption,  given  that  the  process  starts  in  state  i. 

-  -l 

The  first  row  of  (I  -  Q)  ,  which  corresponds  to  starting  in 

state  1,  is  (9/5,  7/5)  from  the  calculations  in  Example  1.  By  counting 

the  number  of  ones  in  the  five  paths  of  Example  1,  we  see  that 

(3  +1+2+2+  1)  =9  entries  into  state  1.  The  average  number  of 

entries  into  state  1  is  therefore  9/5,  which  is  identical  to  the  (1,1) 

-  -1 

entry  of  (I  -  Q)  .  Examining  the  other  transient  state,  we  find  that 
there  are  (1+3+2+  1)  =7  transitions  into  state  2,  for  an  average  of 

7/5  entries  into  state  2  before  absorption.  Since  7/5  is  also  the  (1,2) 

^  -1 

element  of  (I  -  Q)  ,  it  appears  that  another  algebraic  identity,  similar 
to  that  in  the  theorem,  has  been  found.  Further  numerical  experimentation 
leads  to  Theorem  2. 


2 .  Theorem  2 

Under  the  conditions  of  Theorem  1,  the  elements  of  the  first  row  of 
^  -1 

(I  -  Q)  are  identical  to  the  basic  estimates  obtained  directly  from 
the  path  data  by  counting  transitions  into  the  respective  transient  states 
that  is, 


^  q)-'*'  number  of  entries  into  state  j 

1, j  number  of  paths 


(j  =  1,2,...,M) 


a. 


Proof 


The  proof  of  Theorem  2  is  contained  in  Theorem  1.  It  is  only 

necessary  to  note  that  the  ratios  x./x^  (j  =  1,2,...,M)  are  the  mean 

numbers  of  entries  into  transient  states.  These  ratios,  from  Markov 

theory,  equal  the  flow-graph  transmissions  that  in  turn  are  the  elements 

-  -1 

in  the  first  row  of  (I  -  Q)  .  It  can  be  noted  that  Theorems  1  and  2  can 


be  unified  into  a  single  theorem  which  states  that  the  Markov-derived  mean 
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number  of  entries  into  all  states  is  identical  to  the  direct  estimates 
from  the  data;  this  unification  is  possible  because  the  mean  number  of 
entries  into  an  absorption  state  equals  the  probability  of  absorption 
into  that  state. 


b.  Example  of  Theorem  2 

Data  from  the  previous  example  can  be  used  to  illustrate 
Theorem  2.  By  direct  count  from  the  five  paths,  there  are 

(3  +1+2+2+  1)  =9  entries  into  state  1 

(1  +3+2+1)  =7  entries  into  state  2. 

The  first  row  of  (I  -  Q)  is  (9/5,  7/5)  and  thus  verifies  the  identity. 

Theorem  2  concerns  the  mean  number  of  entries  into  each  state 
before  abosrption  while  Theorem  1  concerns  the  absorption  probabilities. 
Theorem  3  is  related  to  another  measure  of  effectiveness:  the  mean  time 
to  absorption  conditioned  on  the  starting  state  and  the  absorbing  state. 


3.  Theorem  3 

Assume  path  data  and  Markov  chain  model  as  in  the  theorem.  Choose 
any  absorbing  state  (say  the  failure  state  f)  and  consider  estimating  the 
mean  time  to  absorption  in  that  state,  conditioned  on  starting  in  state  1 
and  absorption  in  that  state.  Use  only  those  paths  terminating  in  the 
designated  absorbing  state  f  to  estimate  the  p_.  Then  the  Markov  formula 
for  conditional  mean  time  to  absorption  in  state  f,  given  start  in  state 
1,  gives  identically  the  same  mean  time  to  absorb  as  does  the  direct  esti¬ 
mate  from  the  path  data.  Symbolically,  the  (l,f)  element  of 


,  A  -I  A  -h 

(I  -  Q)  (I  -  Q)  R 
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b. 


Example  of  Theorem  3 


Data  from  the  absorption  probability  example  above  will  also 
serve  to  illustrate  the  identity  in  Theorem  3  provided  that  either  state 
3  or  state  4  is  selected  as  an  absorbing  state.  Selecting  state  3  arbi¬ 
trarily,  the  path  data  are 

11123 

12123 

1213 

A  A  /\ 

from  which  P,  Q,  and  R  are  readily  found 


A  —  1 A 

The  absorption  matrix  ( I  -  Q)  R  consists  of  all  ones  because  only  one 
absorbing  state  exists.  Therefore, 


(I  - 


Q)_1( I  - 


A  — «  1  /\ 

Q)  R 


=  (I  -  Q) 


-1 
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The  first  component  (11/3)  agrees  exactly  with  the  mean  number 
determined  from  the  path  data.  (Path  11123  requires  4  transitions,  path 
12123  also  requires  4  transitions,  and  path  1213  requires  3  transitions 
for  a  total  of  eleven  transitions  in  three  trials.)  The  identity  is 
therefore  verified  in  this  case. 
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Appendix  F 


DETERMINING  A  SET  OF  STATES 
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Appendix  F 


DETERMINING  A  SET  OF  STATES 

It  was  indicated  in  Section  XII-C  that  the  choice  of  a  set  of  states 
is  likely  to  be  the  most  difficult  part  of  the  application  of  Markov 
models.  Some  discussion  of  this  problem  aspect  is  therefore  needed. 

Requirements  on  the  state  space  (set  of  states)  imposed  from  the 
mathematical  definition  of  state  are: 

•  States  should  be  defined  to  be  mutually  collective  and  ex¬ 
haustive;  that  is,  at  each  instant  of  time,  the  process 
should  be  in  only  one  state. 

•  The  next  transition  of  the  process  should  depend  only  on  the 
current  state  and  not  on  how  this  state  was  reached. 

The  first  requirement  is  not  difficult  to  achieve  from  the  stand¬ 
point  of  logic  alone.  In  a  complex  realistic  environment,  however,  it 
may  be  difficult  to  ensure  consistency.  Not  much  can  be  said  in  general 
about  this  requirement  except  that  concepts  from  elementary  set  theory 
may  be  useful.  For  example,  if  it  is  desirable  to  have  states  A  and  B 
defined  (for  operational  reasons)  such  that  it  is  possible  to  be  in  both 
states  A  and  B  simultaneously,  then  it  may  be  desirable  to  define  a  new 
state  meaning:  the  process  is  in  both  states  A  and  B.  The  previous 
meaning  of  state  A  must  then  be  modified  to  mean:  in  state  A  and  not 
state  B;  a  similar  modification  would  be  required  for  state  B. 

The  second  requirement — paraphrased  as:  the  future  should  depend 
only  on  the  present  state  and  not  on  the  past  states — is  probably  im¬ 
possible  to  achieve  exactly  in  any  real  application.  When  the  state  is 
search,  for  example,  no  other  information  about  the  previous  states 
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occupied  or  other  variables  is  assumed  to  be  needed  to  determine  the 
future  course  of  the  process. 

In  most  analytical  applications,  the  state  space  is  selected  on  the 
basis  of  information  variables  as  opposed  to  other  possibilities  such  as 
geometrical  variables  (range,  bearing,  speed).  This  will  probably  remain 
true  for  operational  applications  as  well.  However,  the  possibility  of 
using  both  information  and  geometrical  variables  should  be  kept  open 
until  it  is  reasonably  certain  that  geometry  need  not  enter.  To  keep  the 
number  of  states  from  getting  out  hand,  it  may  be  possible  to  introduce 
geometry  by  distinguishing  a  small  number  of  levels  of  important  variables 
such  as  range.  For  example,  a  state  may  be  labeled  by  the  pair  (Search, 
long  range)  to  indicate  that  the  (original)  state  is  Search  and  the  actual 
range  between  the  searcher  and  the  target  is  greater  than  some  specified 
threshold. 

In  applications,  an  analyst's  experience  and  judgment  must  ultimately 
be  used  to  obtain  definitions  for  a  set  of  states.  A  reasonably  systematic 
approach  to  assist  in  state  definition  is  outlined  as  follows. 

•  Determine  a  set  of  dynamic  variables  from  which  states  may  be 

defined.  At  least  two  major  categories  of  variables  will  be 
required:  geometry/kinematic  variables  and  information 

variables. 

•  Choose  a  set  of  states  that  are  mutually  exclusive  and  col¬ 
lectively  exhaustive  and  can  be  determined  from  the  values 
of  the  dynamic  variables. 

•  For  a  selected  set  of  states: 

-  Use  fleet  exercise  data  from  both  exercise  reports  and 
other  historical  data  to  determine  sequence  of  states 
entered  and  the  times  of  entry  into  states.  (The  sequence 
and  associated  times  of  entry  can  be  considered  a  "path" 
through  the  states.) 

-  Perform  tests  to  determine  whether  the  observed  paths  may 
be  assumed  generated  by  a  Markov  or  semi-Markov  process. 
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•  When  the  Markov  assumption  is  justified: 

-  Estimate  parameters  of  the  Markov  model  from  the  path  data, 
using  both  standard  methods  and  methods  in  Section  VI. 

-  Run  the  Markov  model  to  determine  properties  of  the  output. 
Specifically,  determine  the  probability  of  terminating  in 

a  Success  state,  the  mean  time  to  Success  (or  to  Fail), 
and  probability  of  occupying  each  state  as  a  function  of 
time  for  all  states. 
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